visual-layer / fastdup

fastdup is a powerful free tool designed to rapidly extract valuable insights from your image & video datasets. Assisting you to increase your dataset images & labels quality and reduce your data operations costs at an unparalleled scale.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

[Bug]: fastdup fails to create above 10M object crops on ubuntu 20 (due to file system exhaustion)

dbickson opened this issue · comments

What happened?

Storing crop images and subfolder fails to do file system exhaustion on ubuntu 20
Jan 7 00:00:03 ip-172-31-32-254 systemd[1]: Finished Daily man-db regeneration.
Jan 7 00:00:06 ip-172-31-32-254 kernel: [824165.946205] EXT4-fs warning: 110 callbacks suppressed
Jan 7 00:00:06 ip-172-31-32-254 kernel: [824165.946211] EXT4-fs warning (device nvme0n1p1): ext4_dx_add_entry:2544: Directory (ino: 271020531) index full, reach max htree level :2
Jan 7 00:00:06 ip-172-31-32-254 kernel: [824165.946217] EXT4-fs warning (device nvme0n1p1): ext4_dx_add_entry:2548: Large directory feature is not enabled on this filesystem

What did you expect to see?

No response

What version of fastdup were you runnning on?

1.82

What version of Python were you running on?

Python 3.9

Operating System

ubuntu 20

Reproduction steps

No response

Relevant log output

No response

Attach a screenshot [Optional]

No response

Contact Details [Optional]

No response

Fixed in 1.83