visual-layer / fastdup

fastdup is a powerful free tool designed to rapidly extract valuable insights from your image & video datasets. Assisting you to increase your dataset images & labels quality and reduce your data operations costs at an unparalleled scale.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

[Bug]: Kernel keep crashing when trying to run fd.run()

segalinc opened this issue · comments

What happened?

Trying to run a demo using Huggingdaface dataset

Error:
The Kernel crashed while executing code in the current cell or a previous cell. Please review the code in the cell(s) to identify a possible cause of the failure. Click [here](https://aka.ms/vscodeJupyterKernelCrash) for more info. View Jupyter [log](command:jupyter.viewOutput) for further details.

in the log it show it's not able to load onxx models
I also tried with DInoV2 but I get the same issue

I tried both on a terminal and jupiter. On terminal I get segmentation fault

What did you expect to see?

The algorithm running as in the demos

What version of fastdup were you runnning on?

1.85

What version of Python were you running on?

Python 3.10

Operating System

Ubuntut

Reproduction steps

import fastdup
from fastdup.datasets import FastdupHFDataset
dataset = FastdupHFDataset("andrewburns/anime_sprites_v1", split="train", img_key='image', label_key='text')
fd = fastdup.create(input_dir=dataset.img_dir, work_dir='data/fastdup_work_dir') #or dir with images
fd.run(annotations=dataset.annotations, verbose=True)

Relevant log output

FastDup Software, (C) copyright 2022 Dr. Amir Alush and Dr. Danny Bickson.
2024-02-24 19:05:38 [FATAL] Failed to initialize ort model /root/miniforge3/envs/animedm/lib/python3.10/site-packages/fastdup/UndisclosedFastdupModel.ort with error /mnt/data/onnxruntime/onnxruntime/core/platform/posix/env.cc:183 onnxruntime::{anonymous}::PosixThread::PosixThread(const char*, int, unsigned int ()(int, Eigen::ThreadPoolInterface), Eigen::ThreadPoolInterface*, const onnxruntime::ThreadOptions&) pthread_setaffinity_np failed, error code: 2 error msg: No such file or directory

2024-02-24 19:05:38 [ERROR] Failed to initialize ORT

Attach a screenshot [Optional]

image

Contact Details [Optional]

segalin.cristina@gmail.com

Hello @segalinc thanks for reaching out to fastdup support. It seems Jupyter swallows the error message so it is hard to know why there was a crash.
Can you please run the same code on a python terminal / ipython terminal and share here the output you see before the crash? In addition please upgrade fastdup to the latest version using pip3 install -U fastdup.
Thanks

Hello,
I already have the latest version 1.85

this is the error I get running in terminal

fd = fastdup.create(input_dir=dataset.img_dir, work_dir='data/fastdup_work_dir') #or dir with images
>>> fd.run(annotations=dataset.annotations, verbose=True)
FastDup Software, (C) copyright 2022 Dr. Amir Alush and Dr. Danny Bickson.
2024-02-25 04:29:19 [INFO] Version 1.85 Release compiled on Jan 16 2024 09:35:44
2024-02-25 04:29:19 [FATAL] Failed to initialize ort model /root/miniforge3/envs/animedm/lib/python3.10/site-packages/fastdup/UndisclosedFastdupModel.ort with error /mnt/data/onnxruntime/onnxruntime/core/platform/posix/env.cc:183 onnxruntime::{anonymous}::PosixThread::PosixThread(const char*, int, unsigned int (*)(int, Eigen::ThreadPoolInterface*), Eigen::ThreadPoolInterface*, const onnxruntime::ThreadOptions&) pthread_setaffinity_np failed, error code: 2 error msg: No such file or directory

2024-02-25 04:29:19 [ERROR] Failed to initialize ORT
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/root/miniforge3/envs/animedm/lib/python3.10/site-packages/fastdup/engine.py", line 157, in run
    return super().run(annotations=annotations, input_dir=input_dir, subset=subset, data_type=data_type,
  File "/root/miniforge3/envs/animedm/lib/python3.10/site-packages/fastdup/sentry.py", line 144, in inner_function
    raise ex
  File "/root/miniforge3/envs/animedm/lib/python3.10/site-packages/fastdup/sentry.py", line 135, in inner_function
    ret = func(*args, **kwargs)
  File "/root/miniforge3/envs/animedm/lib/python3.10/site-packages/fastdup/fastdup_controller.py", line 584, in run
    raise RuntimeError('Fastdup execution failed')
RuntimeError: Fastdup execution failed
>>> Segmentation fault

Hi @segalinc I see this failed:
Screen Shot 2024-02-25 at 6 34 53
It seems microsoft onnx functionality of setting the number of threads is somehow not supported on your system.
What is your os and which hardware are you running on?

I'm on Ubuntu in an aws ec2 instance

I suspect you are on ubuntu 18 I suggest to upgrade the instance to 20 or 22.
Can you check using
cat /etc/os-release | grep VERSION_ID

yes I am on ubuntu 18 but I can't upgrade I don't have control of the version
does fastdup work only of upgraded versions?

fastdup works fine on ubuntu 18.. I suspect you have an older flavor of ubuntu 18.
We work fine on 18.04

I have 18.04

the error happens even if I change the model

This error will happen for all models since it is an operating system issue, can you check your kernel version using
uname -r

I also tried with a local model

fd.run(model_path='./dinov2_vits14.onnx', annotations=dataset.annotations, verbose=True, d=384)
FastDup Software, (C) copyright 2022 Dr. Amir Alush and Dr. Danny Bickson.
2024-02-25 04:52:58 [INFO] Version 1.85 Release compiled on Jan 16 2024 09:35:44
2024-02-25 04:52:59 [FATAL] Failed to initialize ort model ./dinov2_vits14.onnx with error /mnt/data/onnxruntime/onnxruntime/core/platform/posix/env.cc:183 onnxruntime::{anonymous}::PosixThread::PosixThread(const char*, int, unsigned int (*)(int, Eigen::ThreadPoolInterface*), Eigen::ThreadPoolInterface*, const onnxruntime::ThreadOptions&) pthread_setaffinity_np failed, error code: 2 error msg: No such file or directory

2024-02-25 04:52:59 [ERROR] Failed to initialize ORT
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/root/miniforge3/envs/animedm/lib/python3.10/site-packages/fastdup/engine.py", line 157, in run
    return super().run(annotations=annotations, input_dir=input_dir, subset=subset, data_type=data_type,
  File "/root/miniforge3/envs/animedm/lib/python3.10/site-packages/fastdup/sentry.py", line 144, in inner_function
    raise ex
  File "/root/miniforge3/envs/animedm/lib/python3.10/site-packages/fastdup/sentry.py", line 135, in inner_function
Segmentation fault

Our kernel version which works fine is 5.4.0-1106-gcp
Your kernel is 5.15.134, need to check why pthread_setaffinity_np fails..

I suggest contacting your admins asking for a newer OS build...

could this be related? microsoft/onnxruntime#10113 ?
Admins won't change the build of a company for this eheh

I tried something and this is what I got

>>> import onnxruntime
>>> onnxruntime.InferenceSession("dinov2_vits14.onnx")
2024-02-25 05:13:52.685580921 [E:onnxruntime:Default, env.cc:254 ThreadMain] pthread_setaffinity_np failed for thread: 1349128, index: 0, mask: {0, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-02-25 05:13:52.685752673 [E:onnxruntime:Default, env.cc:254 ThreadMain] pthread_setaffinity_np failed for thread: 1349130, index: 2, mask: {1, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-02-25 05:13:52.685855664 [E:onnxruntime:Default, env.cc:254 ThreadMain] pthread_setaffinity_np failed for thread: 1349132, index: 4, mask: {2, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-02-25 05:13:52.685980576 [E:onnxruntime:Default, env.cc:254 ThreadMain] pthread_setaffinity_np failed for thread: 1349134, index: 6, mask: {3, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-02-25 05:13:52.686135718 [E:onnxruntime:Default, env.cc:254 ThreadMain] pthread_setaffinity_np failed for thread: 1349136, index: 8, mask: {4, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-02-25 05:13:52.686207999 [E:onnxruntime:Default, env.cc:254 ThreadMain] pthread_setaffinity_np failed for thread: 1349138, index: 10, mask: {5, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-02-25 05:13:52.686326860 [E:onnxruntime:Default, env.cc:254 ThreadMain] pthread_setaffinity_np failed for thread: 1349140, index: 12, mask: {6, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-02-25 05:13:52.686427871 [E:onnxruntime:Default, env.cc:254 ThreadMain] pthread_setaffinity_np failed for thread: 1349142, index: 14, mask: {7, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-02-25 05:13:52.686566383 [E:onnxruntime:Default, env.cc:254 ThreadMain] pthread_setaffinity_np failed for thread: 1349144, index: 16, mask: {8, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-02-25 05:13:52.686662324 [E:onnxruntime:Default, env.cc:254 ThreadMain] pthread_setaffinity_np failed for thread: 1349146, index: 18, mask: {9, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-02-25 05:13:52.686780725 [E:onnxruntime:Default, env.cc:254 ThreadMain] pthread_setaffinity_np failed for thread: 1349148, index: 20, mask: {10, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-02-25 05:13:52.686915507 [E:onnxruntime:Default, env.cc:254 ThreadMain] pthread_setaffinity_np failed for thread: 1349150, index: 22, mask: {11, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-02-25 05:13:52.687007318 [E:onnxruntime:Default, env.cc:254 ThreadMain] pthread_setaffinity_np failed for thread: 1349152, index: 24, mask: {12, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-02-25 05:13:52.687112709 [E:onnxruntime:Default, env.cc:254 ThreadMain] pthread_setaffinity_np failed for thread: 1349154, index: 26, mask: {13, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-02-25 05:13:52.687210470 [E:onnxruntime:Default, env.cc:254 ThreadMain] pthread_setaffinity_np failed for thread: 1349156, index: 28, mask: {14, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-02-25 05:13:52.687316701 [E:onnxruntime:Default, env.cc:254 ThreadMain] pthread_setaffinity_np failed for thread: 1349158, index: 30, mask: {15, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-02-25 05:13:52.687412752 [E:onnxruntime:Default, env.cc:254 ThreadMain] pthread_setaffinity_np failed for thread: 1349160, index: 32, mask: {16, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-02-25 05:13:52.687519934 [E:onnxruntime:Default, env.cc:254 ThreadMain] pthread_setaffinity_np failed for thread: 1349162, index: 34, mask: {17, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-02-25 05:13:52.687646225 [E:onnxruntime:Default, env.cc:254 ThreadMain] pthread_setaffinity_np failed for thread: 1349164, index: 36, mask: {18, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-02-25 05:13:52.687801807 [E:onnxruntime:Default, env.cc:254 ThreadMain] pthread_setaffinity_np failed for thread: 1349166, index: 38, mask: {19, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-02-25 05:13:52.687892548 [E:onnxruntime:Default, env.cc:254 ThreadMain] pthread_setaffinity_np failed for thread: 1349168, index: 40, mask: {20, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-02-25 05:13:52.687994269 [E:onnxruntime:Default, env.cc:254 ThreadMain] pthread_setaffinity_np failed for thread: 1349170, index: 42, mask: {21, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-02-25 05:13:52.688087520 [E:onnxruntime:Default, env.cc:254 ThreadMain] pthread_setaffinity_np failed for thread: 1349172, index: 44, mask: {22, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-02-25 05:13:52.688176901 [E:onnxruntime:Default, env.cc:254 ThreadMain] pthread_setaffinity_np failed for thread: 1349174, index: 46, mask: {23, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-02-25 05:13:52.688277643 [E:onnxruntime:Default, env.cc:254 ThreadMain] pthread_setaffinity_np failed for thread: 1349176, index: 48, mask: {24, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-02-25 05:13:52.688363384 [E:onnxruntime:Default, env.cc:254 ThreadMain] pthread_setaffinity_np failed for thread: 1349178, index: 50, mask: {25, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-02-25 05:13:52.688477245 [E:onnxruntime:Default, env.cc:254 ThreadMain] pthread_setaffinity_np failed for thread: 1349180, index: 52, mask: {26, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-02-25 05:13:52.688589806 [E:onnxruntime:Default, env.cc:254 ThreadMain] pthread_setaffinity_np failed for thread: 1349182, index: 54, mask: {27, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-02-25 05:13:52.688693007 [E:onnxruntime:Default, env.cc:254 ThreadMain] pthread_setaffinity_np failed for thread: 1349184, index: 56, mask: {28, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-02-25 05:13:52.688792769 [E:onnxruntime:Default, env.cc:254 ThreadMain] pthread_setaffinity_np failed for thread: 1349186, index: 58, mask: {29, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-02-25 05:13:52.688898040 [E:onnxruntime:Default, env.cc:254 ThreadMain] pthread_setaffinity_np failed for thread: 1349188, index: 60, mask: {30, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-02-25 05:13:52.688983781 [E:onnxruntime:Default, env.cc:254 ThreadMain] pthread_setaffinity_np failed for thread: 1349190, index: 62, mask: {31, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-02-25 05:13:52.689125972 [E:onnxruntime:Default, env.cc:254 ThreadMain] pthread_setaffinity_np failed for thread: 1349192, index: 64, mask: {32, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-02-25 05:13:52.689264584 [E:onnxruntime:Default, env.cc:254 ThreadMain] pthread_setaffinity_np failed for thread: 1349194, index: 66, mask: {33, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-02-25 05:13:52.689385705 [E:onnxruntime:Default, env.cc:254 ThreadMain] pthread_setaffinity_np failed for thread: 1349196, index: 68, mask: {34, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-02-25 05:13:52.689517587 [E:onnxruntime:Default, env.cc:254 ThreadMain] pthread_setaffinity_np failed for thread: 1349198, index: 70, mask: {35, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-02-25 05:13:52.689626078 [E:onnxruntime:Default, env.cc:254 ThreadMain] pthread_setaffinity_np failed for thread: 1349200, index: 72, mask: {36, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-02-25 05:13:52.689740439 [E:onnxruntime:Default, env.cc:254 ThreadMain] pthread_setaffinity_np failed for thread: 1349202, index: 74, mask: {37, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-02-25 05:13:52.689840311 [E:onnxruntime:Default, env.cc:254 ThreadMain] pthread_setaffinity_np failed for thread: 1349204, index: 76, mask: {38, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-02-25 05:13:52.689945892 [E:onnxruntime:Default, env.cc:254 ThreadMain] pthread_setaffinity_np failed for thread: 1349206, index: 78, mask: {39, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-02-25 05:13:52.690085773 [E:onnxruntime:Default, env.cc:254 ThreadMain] pthread_setaffinity_np failed for thread: 1349208, index: 80, mask: {40, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-02-25 05:13:52.690216565 [E:onnxruntime:Default, env.cc:254 ThreadMain] pthread_setaffinity_np failed for thread: 1349210, index: 82, mask: {41, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-02-25 05:13:52.690313866 [E:onnxruntime:Default, env.cc:254 ThreadMain] pthread_setaffinity_np failed for thread: 1349212, index: 84, mask: {42, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-02-25 05:13:52.690423507 [E:onnxruntime:Default, env.cc:254 ThreadMain] pthread_setaffinity_np failed for thread: 1349214, index: 86, mask: {43, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-02-25 05:13:52.690571989 [E:onnxruntime:Default, env.cc:254 ThreadMain] pthread_setaffinity_np failed for thread: 1349216, index: 88, mask: {44, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-02-25 05:13:52.690684870 [E:onnxruntime:Default, env.cc:254 ThreadMain] pthread_setaffinity_np failed for thread: 1349218, index: 90, mask: {45, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-02-25 05:13:52.690805822 [E:onnxruntime:Default, env.cc:254 ThreadMain] pthread_setaffinity_np failed for thread: 1349220, index: 92, mask: {46, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
2024-02-25 05:13:52.690912723 [E:onnxruntime:Default, env.cc:254 ThreadMain] pthread_setaffinity_np failed for thread: 1349222, index: 94, mask: {47, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
<onnxruntime.capi.onnxruntime_inference_collection.InferenceSession object at 0x7f0d2c8194b0>```

Hi @segalinc we can try to compile a version for ubuntu 18 which avoids setting up the threads in ONNX, however it may require your patience to test it...

HI @segalinc I found the issue, I suddenly remembered this one: #300.
Onnxruntime is trying to set the affinity of a thread into a specific cpu, this failed in issue 300 on docker virtualization, I believe your ubuntu machine may be a kubernetis node running within docker. We have compiled v 1.100 for Ubutnu 18 skipping setting the affinity. Please try it out and let us know if this works.

I gave it a try
when running I get
Error: failed to set cpu affinity error code 2 No such file or directory multiple times but then it seems working (i added verbose=True)
Thanks so much for digging into this
Very excited to use the library now !

Thanks @segalinc for your quick report! Indeed I changed the error into a warning, you can safely ignore it.