Error verifying pickled file

Question

Error verifying pickled file

INTstinkt opened this issue 6 months ago · comments

*** Error verifying pickled file from C:\AI\stable-diffusion-webui\extensions\OneButtonPrompt\superprompter./model_files/pytorch_model.bin
*** The file may be malicious, so the program is not going to read it.
*** You can skip this check with --disable-safe-unpickle commandline argument.

Traceback (most recent call last):
  File "C:\AI\stable-diffusion-webui\modules\safe.py", line 137, in load_with_extra
    check_pt(filename, extra_handler)
  File "C:\AI\stable-diffusion-webui\modules\safe.py", line 84, in check_pt
    check_zip_filenames(filename, z.namelist())
  File "C:\AI\stable-diffusion-webui\modules\safe.py", line 76, in check_zip_filenames
    raise Exception(f"bad file inside {filename}: {name}")
Exception: bad file inside C:\AI\stable-diffusion-webui\extensions\OneButtonPrompt\superprompter\./model_files/pytorch_model.bin: pytorch_model/byteorder

*** Error completing request
*** Arguments: ('task(jctrfbn3car3sqr)', <gradio.routes.Request object at 0x0000027BE3B23A00>, '', '', [], 20, 'DPM++ 2M Karras', 1, 1, 7, 1024, 1024, False, 0.7, 2, 'Latent', 0, 0, 0, 'Use same checkpoint', 'Use same sampler', '', '', [], 4, False, '', 0.8, -1, False, -1, 0, 0, 0, False, False, {'ad_model': 'face_yolov8n.pt', 'ad_model_classes': '', 'ad_prompt': '', 'ad_negative_prompt': '', 'ad_confidence': 0.3, 'ad_mask_k_largest': 0, 'ad_mask_min_ratio': 0, 'ad_mask_max_ratio': 1, 'ad_x_offset': 0, 'ad_y_offset': 0, 'ad_dilate_erode': 4, 'ad_mask_merge_invert': 'None', 'ad_mask_blur': 4, 'ad_denoising_strength': 0.4, 'ad_inpaint_only_masked': True, 'ad_inpaint_only_masked_padding': 32, 'ad_use_inpaint_width_height': False, 'ad_inpaint_width': 512, 'ad_inpaint_height': 512, 'ad_use_steps': False, 'ad_steps': 28, 'ad_use_cfg_scale': False, 'ad_cfg_scale': 7, 'ad_use_checkpoint': False, 'ad_checkpoint': 'Use same checkpoint', 'ad_use_vae': False, 'ad_vae': 'Use same VAE', 'ad_use_sampler': False, 'ad_sampler': 'Euler a', 'ad_use_noise_multiplier': False, 'ad_noise_multiplier': 1, 'ad_use_clip_skip': False, 'ad_clip_skip': 1, 'ad_restore_face': False, 'ad_controlnet_model': 'None', 'ad_controlnet_module': 'None', 'ad_controlnet_weight': 1, 'ad_controlnet_guidance_start': 0, 'ad_controlnet_guidance_end': 1, 'is_api': ()}, {'ad_model': 'None', 'ad_model_classes': '', 'ad_prompt': '', 'ad_negative_prompt': '', 'ad_confidence': 0.3, 'ad_mask_k_largest': 0, 'ad_mask_min_ratio': 0, 'ad_mask_max_ratio': 1, 'ad_x_offset': 0, 'ad_y_offset': 0, 'ad_dilate_erode': 4, 'ad_mask_merge_invert': 'None', 'ad_mask_blur': 4, 'ad_denoising_strength': 0.4, 'ad_inpaint_only_masked': True, 'ad_inpaint_only_masked_padding': 32, 'ad_use_inpaint_width_height': False, 'ad_inpaint_width': 512, 'ad_inpaint_height': 512, 'ad_use_steps': False, 'ad_steps': 28, 'ad_use_cfg_scale': False, 'ad_cfg_scale': 7, 'ad_use_checkpoint': False, 'ad_checkpoint': 'Use same checkpoint', 'ad_use_vae': False, 'ad_vae': 'Use same VAE', 'ad_use_sampler': False, 'ad_sampler': 'Euler a', 'ad_use_noise_multiplier': False, 'ad_noise_multiplier': 1, 'ad_use_clip_skip': False, 'ad_clip_skip': 1, 'ad_restore_face': False, 'ad_controlnet_model': 'None', 'ad_controlnet_module': 'None', 'ad_controlnet_weight': 1, 'ad_controlnet_guidance_start': 0, 'ad_controlnet_guidance_end': 1, 'is_api': ()}, {'ad_model': 'None', 'ad_model_classes': '', 'ad_prompt': '', 'ad_negative_prompt': '', 'ad_confidence': 0.3, 'ad_mask_k_largest': 0, 'ad_mask_min_ratio': 0, 'ad_mask_max_ratio': 1, 'ad_x_offset': 0, 'ad_y_offset': 0, 'ad_dilate_erode': 4, 'ad_mask_merge_invert': 'None', 'ad_mask_blur': 4, 'ad_denoising_strength': 0.4, 'ad_inpaint_only_masked': True, 'ad_inpaint_only_masked_padding': 32, 'ad_use_inpaint_width_height': False, 'ad_inpaint_width': 512, 'ad_inpaint_height': 512, 'ad_use_steps': False, 'ad_steps': 28, 'ad_use_cfg_scale': False, 'ad_cfg_scale': 7, 'ad_use_checkpoint': False, 'ad_checkpoint': 'Use same checkpoint', 'ad_use_vae': False, 'ad_vae': 'Use same VAE', 'ad_use_sampler': False, 'ad_sampler': 'DPM++ 2M Karras', 'ad_use_noise_multiplier': False, 'ad_noise_multiplier': 1, 'ad_use_clip_skip': False, 'ad_clip_skip': 1, 'ad_restore_face': False, 'ad_controlnet_model': 'None', 'ad_controlnet_module': 'None', 'ad_controlnet_weight': 1, 'ad_controlnet_guidance_start': 0, 'ad_controlnet_guidance_end': 1, 'is_api': ()}, {'ad_model': 'None', 'ad_model_classes': '', 'ad_prompt': '', 'ad_negative_prompt': '', 'ad_confidence': 0.3, 'ad_mask_k_largest': 0, 'ad_mask_min_ratio': 0, 'ad_mask_max_ratio': 1, 'ad_x_offset': 0, 'ad_y_offset': 0, 'ad_dilate_erode': 4, 'ad_mask_merge_invert': 'None', 'ad_mask_blur': 4, 'ad_denoising_strength': 0.4, 'ad_inpaint_only_masked': True, 'ad_inpaint_only_masked_padding': 32, 'ad_use_inpaint_width_height': False, 'ad_inpaint_width': 512, 'ad_inpaint_height': 512, 'ad_use_steps': False, 'ad_steps': 28, 'ad_use_cfg_scale': False, 'ad_cfg_scale': 7, 'ad_use_checkpoint': False, 'ad_checkpoint': 'Use same checkpoint', 'ad_use_vae': False, 'ad_vae': 'Use same VAE', 'ad_use_sampler': False, 'ad_sampler': 'DPM++ 2M Karras', 'ad_use_noise_multiplier': False, 'ad_noise_multiplier': 1, 'ad_use_clip_skip': False, 'ad_clip_skip': 1, 'ad_restore_face': False, 'ad_controlnet_model': 'None', 'ad_controlnet_module': 'None', 'ad_controlnet_weight': 1, 'ad_controlnet_guidance_start': 0, 'ad_controlnet_guidance_end': 1, 'is_api': ()}, False, 'MultiDiffusion', False, True, 1024, 1024, 96, 96, 48, 4, 'None', 2, False, 10, 1, 1, 64, False, False, False, False, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 'DemoFusion', False, 128, 64, 4, 2, False, 10, 1, 1, 64, False, True, 3, 1, 1, True, 0.85, 0.5, 4, False, False, 3072, 192, True, True, True, False, <scripts.animatediff_ui.AnimateDiffProcess object at 0x0000027BE3B22E60>, False, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, False, -1, -1, 0, 0, '1,1', 'Horizontal', '', 2, 1, [], UiControlNetUnit(enabled=False, module='none', model='None', weight=1, image=None, resize_mode='Crop and Resize', low_vram=False, processor_res=-1, threshold_a=-1, threshold_b=-1, guidance_start=0, guidance_end=1, pixel_perfect=False, control_mode='Balanced', inpaint_crop_input_image=False, hr_option='Both', save_detected_map=True, advanced_weighting=None), UiControlNetUnit(enabled=False, module='none', model='None', weight=1, image=None, resize_mode='Crop and Resize', low_vram=False, processor_res=-1, threshold_a=-1, threshold_b=-1, guidance_start=0, guidance_end=1, pixel_perfect=False, control_mode='Balanced', inpaint_crop_input_image=False, hr_option='Both', save_detected_map=True, advanced_weighting=None), UiControlNetUnit(enabled=False, module='none', model='None', weight=1, image=None, resize_mode='Crop and Resize', low_vram=False, processor_res=-1, threshold_a=-1, threshold_b=-1, guidance_start=0, guidance_end=1, pixel_perfect=False, control_mode='Balanced', inpaint_crop_input_image=False, hr_option='Both', save_detected_map=True, advanced_weighting=None), UiControlNetUnit(enabled=False, module='none', model='None', weight=1, image=None, resize_mode='Crop and Resize', low_vram=False, processor_res=-1, threshold_a=-1, threshold_b=-1, guidance_start=0, guidance_end=1, pixel_perfect=False, control_mode='Balanced', inpaint_crop_input_image=False, hr_option='Both', save_detected_map=True, advanced_weighting=None), UiControlNetUnit(enabled=False, module='none', model='None', weight=1, image=None, resize_mode='Crop and Resize', low_vram=False, processor_res=-1, threshold_a=-1, threshold_b=-1, guidance_start=0, guidance_end=1, pixel_perfect=False, control_mode='Balanced', inpaint_crop_input_image=False, hr_option='Both', save_detected_map=True, advanced_weighting=None), UiControlNetUnit(enabled=False, module='none', model='None', weight=1, image=None, resize_mode='Crop and Resize', low_vram=False, processor_res=-1, threshold_a=-1, threshold_b=-1, guidance_start=0, guidance_end=1, pixel_perfect=False, control_mode='Balanced', inpaint_crop_input_image=False, hr_option='Both', save_detected_map=True, advanced_weighting=None), UiControlNetUnit(enabled=False, module='none', model='None', weight=1, image=None, resize_mode='Crop and Resize', low_vram=False, processor_res=-1, threshold_a=-1, threshold_b=-1, guidance_start=0, guidance_end=1, pixel_perfect=False, control_mode='Balanced', inpaint_crop_input_image=False, hr_option='Both', save_detected_map=True, advanced_weighting=None), 0, False, 'Default', 'Default', 1, False, 0, False, 0, 0, 0, 0, False, False, 3, 0, False, False, 0, False, False, False, 'positive', 'comma', 0, False, False, 'start', '', 1, '', [], 0, '', [], 0, '', [], True, False, False, False, False, False, False, 0, False, 5, 'all', 'all', 'all', '', '', '', '1', 'none', False, '', '', 'comma', '', True, '', '20', 'all', 'all', 'all', 'all', 0, '', True, '0', False, 'SDXL', 'Standard', 'dynamic', 'superprompt-v1', None, None, False, None, None, False, None, None, False, None, None, False, None, None, False, None, None, False, None, None, False, 50, 20, 0.55, 'ESRGAN_4x', 'Lightning\dreamshaperXL_lightningDPMSDE.safetensors [fdbe56354b]', True, False, 0, 'Range', 1, 'GPU', True, False, False, False, False, False, 0, 448, False, 448, False, False, 3, False, 3, True, 3, False, 'Horizontal', False, False, 'u2net', False, True, True, False, 0, 2.5, 'polylines_sharp', ['left-right', 'red-cyan-anaglyph'], 2, 0, '∯boost∯clipdepth∯clipdepth_far∯clipdepth_mode∯clipdepth_near∯compute_device∯do_output_depth∯gen_heatmap∯gen_normalmap∯gen_rembg∯gen_simple_mesh∯gen_stereo∯model_type∯net_height∯net_size_match∯net_width∯normalmap_invert∯normalmap_post_blur∯normalmap_post_blur_kernel∯normalmap_pre_blur∯normalmap_pre_blur_kernel∯normalmap_sobel∯normalmap_sobel_kernel∯output_depth_combine∯output_depth_combine_axis∯output_depth_invert∯pre_depth_background_removal∯rembg_model∯save_background_removal_masks∯save_outputs∯simple_mesh_occlude∯simple_mesh_spherical∯stereo_balance∯stereo_divergence∯stereo_fill_algo∯stereo_modes∯stereo_offset_exponent∯stereo_separation', 'Positive', 0, ', ', 'Generate and always save', 32) {}
Traceback (most recent call last):
File "C:\AI\stable-diffusion-webui\modules\call_queue.py", line 57, in f
res = list(func(*args, **kwargs))
File "C:\AI\stable-diffusion-webui\modules\call_queue.py", line 36, in f
res = func(*args, **kwargs)
File "C:\AI\stable-diffusion-webui\modules\txt2img.py", line 107, in txt2img
processed = modules.scripts.scripts_txt2img.run(p, *p.script_args)
File "C:\AI\stable-diffusion-webui\modules\scripts.py", line 766, in run
processed = script.run(p, *script_args)
File "C:\AI\stable-diffusion-webui\extensions\OneButtonPrompt\scripts\onebuttonprompt.py", line 1347, in run
base_prompt = build_dynamic_prompt(insanitylevel,subject,artist, imagetype, False, antistring, prefixprompt, suffixprompt,promptcompounderlevel, seperator,givensubject,smartsubject,giventypeofimage,imagemodechance,chosengender, chosensubjectsubtypeobject, chosensubjectsubtypehumanoid, chosensubjectsubtypeconcept,True,False,-1,givenoutfit, False, base_model, OBP_preset, promptenhancer)
File "C:\AI\stable-diffusion-webui\extensions\OneButtonPrompt\build_dynamic_prompt.py", line 3312, in build_dynamic_prompt
superpromptresult = one_button_superprompt(insanitylevel=insanitylevel, prompt=subjectprompt, seed=seed, override_subject=givensubject, override_outfit=overrideoutfit, chosensubject=subjectchooser, gender=gender, restofprompt = startprompt + endprompt)
File "C:\AI\stable-diffusion-webui\extensions\OneButtonPrompt\build_dynamic_prompt.py", line 4746, in one_button_superprompt
load_models()
File "C:\AI\stable-diffusion-webui\extensions\OneButtonPrompt\superprompter\superprompter.py", line 25, in load_models
model = T5ForConditionalGeneration.from_pretrained(modelDir, torch_dtype=torch.float16)
File "C:\AI\stable-diffusion-webui\venv\lib\site-packages\transformers\modeling_utils.py", line 2657, in from_pretrained
loaded_state_dict_keys = list(state_dict.keys())
AttributeError: 'NoneType' object has no attribute 'keys'

AIrjen · Answer 1 · Thu Apr 11 2024 01:10:44 GMT+0800 (China Standard Time)

Thanks for reporting! It might be something to do with the OS? I can see a dot in the directory name which shouldn't be there (after superprompter):
C:\AI\stable-diffusion-webui\extensions\OneButtonPrompt\superprompter./model_files/pytorch_model.bin

I can take a look in the weekend to see if I can resolve this. If you have any OS information for me, that might help.

INTstinkt · Answer 2 · Fri Apr 12 2024 13:50:59 GMT+0800 (China Standard Time)

Windows 10 Pro Version 22H2

AIrjen · Answer 3 · Sat Apr 13 2024 16:30:14 GMT+0800 (China Standard Time)

I have spend some time checking for this error. I don't have a concrete answer.
The method of checking in WebUI hasn't changed since last year (https://github.com/AUTOMATIC1111/stable-diffusion-webui/blob/master/modules/safe.py)

Maybe some parts of your WebUI is old, or out of date?

I would suggest the following steps, and trying these in order:

Delete the /model_files/ folder in the OneButtonPrompt\superprompter\ folder. This will redownload on first use again.
Make sure all the requirements are installed, Go to OneButtonPrompt\superprompter\ and pip install requirements.txt
Update WebUI to the latest version
Complete reinstall of WebUI

If that all doesn't work, then just add "--disable-safe-unpickle" to your startup command line. That should at least keep WebUI from throwing the error.

INTstinkt · Answer 4 · Sat Apr 20 2024 13:01:53 GMT+0800 (China Standard Time)

Ok A1111 was completely updated, all extensions also up to date.
i deleted the folder u mentioned went to check if i have to pip install the requirements in extensions\OneButtonPrompt\superprompter and for whatever reasons i am getting this weird notification

C:\AI\stable-diffusion-webui\extensions\OneButtonPrompt\superprompter>pip install requirements.txt
ERROR: Could not find a version that satisfies the requirement requirements.txt (from versions: none)
HINT: You are attempting to install a package literally named "requirements.txt" (which cannot exist). Consider using the '-r' flag to install the packages listed in requirements.txt
ERROR: No matching distribution found for requirements.txt

when im opening up the requirements.txt this is inside:
transformers
einops
accelerate
sentencepiece

i then went on fresh installed newest a1111 seperat only downloaded the extension and got the same error over again.

really not feeling like throwing that command arg in there since it kind of bears some risks no?

EDIT: Funny enough i just checked if it would also be the same on my installation of forge and in there i have no problems it just worked there but i dont use forge at all anymore

AIrjen · Answer 5 · Sat May 04 2024 15:25:03 GMT+0800 (China Standard Time)

Hi! Sorry, for the late reaction.

The error is right, the command should be pip install -r requirements.txt. It looks like some of the requirements might be missing in the virtual environment of A1111 (but do exists on forge).

I'll update the documentation to add the -r

Hope that helps, not much I can do from here :(

Wolbis · Answer 6 · Sat May 18 2024 16:19:12 GMT+0800 (China Standard Time)

I'm getting this issue too. :/

I'm on Windows 11 Pro and my WebUI is up to date.

Would love to try this feature out but I'm not really comfortable disabling the unsafe pickle check. :(

INTstinkt · Answer 7 · Sun May 19 2024 10:16:45 GMT+0800 (China Standard Time)

sorry to hear that, i just accepted the fact that its not working for me i pip ibstalled all requirements like author told me to but after i still got same error so i didnt try any further sorry i couldnt help:( maybe if you use forge or comfy it works there, for me it works everywhere except A1111. would also not recommend to disable unsafe pickle

AndreRatzenberger · Answer 8 · Thu Jun 06 2024 08:22:10 GMT+0800 (China Standard Time)

Replace the model.bin with the model.safetensors of this repo https://huggingface.co/roborovski/superprompt-v1

seems to work!

INTstinkt · Answer 9 · Thu Jun 06 2024 12:52:41 GMT+0800 (China Standard Time)

wow this actually did work! thank you very much.

Wolbis · Answer 10 · Thu Jun 06 2024 14:31:21 GMT+0800 (China Standard Time)

Awesome! Thank you, AndreRatzenberger. Would have been nice if the creator of the extension could have pointed us to this.

AIrjen · Answer 11 · Fri Jun 07 2024 02:29:19 GMT+0800 (China Standard Time)

Wow, that is awesome! Thanks, I will try and look if I can make this part of the normal flow!

AIrjen · Answer 12 · Fri Jun 07 2024 15:04:56 GMT+0800 (China Standard Time)

I think I know why this is happening. On my machine it downloads the .safetensor file by itself, and not the .bin file.

I have been reading through the docs of the tokenizer, and here it states the following:

So, what I think it happening, is that the safetensors package isn't installed, and that is why it defaults to downloading the .bin file instead.

So the theoretical fix is now:

I am adding safetensors into the requirements.txt file in the next release, to prevent issues in the future.
I will add this in the docs, to make it more explicit.
on your environment, do a pip install safetensors, and remove the folder to retrigger the download. And check wether it now takes the safetensor file.