lijiunderstand / YFCC15M_downloader

A subset of YFCC100M. Tools, checking scripts and links of web drive to download datasets(uncompressed).

YFCC15M_downloader

A subset of YFCC100M. Tools, checking scripts and links of web drive to download datasets.

---

We followed the dataset preparation process of DeCLIP here.

1. First, Download DeCLIP's YFCC15M label file 'yfcc15m_clean_open_data.json' at Google Driver.

2. Extract the URL from the JSON file and split it into several URL list files for download using split_download_task.py.

3. Crawl the image by the URL dirctely using auto_download.bat (Here, we use Wget, you may need to install that). The bat file is for Windows, and you may need to rewrite a shell file if using Linux. Or, simply download from the links below!

   • You can stop the process and start over afterward if something is wrong. Wget will skip the downloaded files and clean log files.
   • The error will be recorded in log files. Before re-start the download, it is recommended to run clean_err_file_from_logs.py to filter and delete the wrong files.

4. Check the downloaded images using check_images.py.

---

Dataset infos:

• The dataset should contains 15,388,848 images.
• We managed to crawl 15,061,747 of them.
• Total space occupied: 867.73G.

Web Drive links:

• 📂split_1✅
• 📂split_2✅
• 📂split_3✅
• 📂split_4✅
• 📂split_5✅
• 📂split_6✅
• 📂split_7✅
• 📂split_8✅
• 📂split_9✅
• 📂split_10✅
• 📂split_11✅
• 📂split_12✅
• 📂split_13✅
• 📂split_14✅
• 📂split_15✅
• 📂split_16✅

---

If the link fails, please leave a message in the issue.