php-fpm crashing when downloading big folder
ouafnico opened this issue · comments
Detailed description of the problem
When you open a big folder, with a lot of pictures (in my case 500), when you click the download button, the server start to create a zip file.
php-fpm is using one cpu, and after the "max_execution_time" on config, will crash.
You get a 503 error on browser.
Steps to reproduce the issue
Create a big folder with a lot of pictures and try to download the entire album.
Output of the diagnostics (Settings => Diagnostics)
Diagnostics
Warning: Dropbox import not working. No property for dropboxKey.
No critical problems found. Lychee should work without problems!
System Information
Lychee Version (json): 3.2.16
Lychee Version (git): cbd6730 (master)
DB Version: update_030216
System: Linux
PHP Version: 7.3
MySQL Version: 50505
Imagick: 1
Imagick Active: 1
Imagick Version: 1687
GD Version: 2.2.5
Plugins:
Config Information
checkForUpdates: 1
default_license: CC-BY-NC-ND
deleteImported: 1
full_photo: 1
hide_version_number: 1
image_overlay: 0
image_overlay_type: exif
imagick: 1
lang: fr
layout: 1
medium_max_height: 1080
medium_max_width: 1920
php_script_limit: 0
public_search: 0
skipDuplicates: 0
small_max_height: 360
small_max_width: 0
sortingAlbums: ORDER BY title ASC
sortingPhotos: ORDER BY takestamp ASC
useExiftool: 0
version: update_030216
Browser and system
Chrome last version
Linux/Windows (both same problem)
Any way to create the zip file manually before?
- Yes, this is a known issue (but it has been lost with the migration from electerious to LycheeOrg).
- Yes, this has been fixed... in the v4 available here: https://github.com/LycheeOrg/Lychee-laravel
- No, this won't be fixed in the v3 unless someone makes a PR for it.
Related:
LycheeOrg/Lychee#253
LycheeOrg/Lychee#17
Thanks
So I need to use the https://github.com/LycheeOrg/Lychee-laravel version ?
The lychee-laravel version will be the v4 and should be released soon™.
So if you want to give a try to the migration, please do. It is non destructive.
A bit of documentation: https://github.com/LycheeOrg/Lychee-Laravel/wiki/Install (still need to be improved but most things are here).
Basically, you clone the repo with --recurse-submodules, copy/move your upload folder to public and then do the composer and migrate steps... :)
@ildyria Is this actually fixed? I mean, I know that we now create the ZIP file as-we-go, while streaming to the client, but presumably for a large enough folder we could still exceed the execution time limit, leaving the client with a partial zip file?
Of course, if this is the case, v4 does provide a semi-workable workaround of multiselecting a range of images and downloading them that way, say, a hundred at a time...
Actually I can't try the v4. On debian 9 with php 7.3 composer is failing during install deps.
Is this actually fixed? I mean, I know that we now create the ZIP file as-we-go, while streaming to the client, but presumably for a large enough folder we could still exceed the execution time limit, leaving the client with a partial zip file?
To be checked.
Actually I can't try the v4. On debian 9 with php 7.3 composer is failing during install deps.
Could you open an issue there : https://github.com/LycheeOrg/Lychee-laravel
Because as far as I know, this is being checked by the CI so this should not fail.
https://github.com/LycheeOrg/Lychee-Laravel/wiki/Install-Steps-on-Debian-(Testing) might also be worth reading. :)
Actually I can't try the v4. On debian 9 with php 7.3 composer is failing during install deps.
Could you open an issue there : https://github.com/LycheeOrg/Lychee-laravel
Because as far as I know, this is being checked by the CI so this should not fail.https://github.com/LycheeOrg/Lychee-Laravel/wiki/Install-Steps-on-Debian-(Testing) might also be worth reading. :)
Done : LycheeOrg/Lychee#310
I think it's a composer problem, not Lychee's.
The lychee-laravel version will be the v4 and should be released soon™.
So if you want to give a try to the migration, please do. It is non destructive.A bit of documentation: https://github.com/LycheeOrg/Lychee-Laravel/wiki/Install (still need to be improved but most things are here).
Basically, you clone the repo with--recurse-submodules, copy/move youruploadfolder topublicand then do thecomposerandmigratesteps... :)
Good upgrade works.
I just got some errors on logs:
2019-07-26 15:57:00 -- error -- App\Configs::get_value -- 167 -- checkForUpdates does not exist in config (local) !
2019-07-26 15:56:16 -- error -- App\Configs::get_value -- 167 -- full_photo does not exist in config (local) !
2019-07-26 15:56:09 -- error -- App\Configs::get_value -- 167 -- imagick does not exist in config (local) !
2019-07-26 15:56:09 -- error -- App\Configs::get_value -- 167 -- compression_quality does not exist in config (local) !
2019-07-26 15:56:06 -- error -- App\Configs::get_value -- 167 -- imagick does not exist in config (local) !
2019-07-26 15:56:06 -- error -- App\Configs::get_value -- 167 -- compression_quality does not exist in config (local) !
2019-07-26 15:55:58 -- error -- App\Configs::get_value -- 167 -- imagick does not exist in config (local) !
2019-07-26 15:55:58 -- error -- App\Configs::get_value -- 167 -- compression_quality does not exist in config (local) !
2019-07-26 15:51:28 -- error -- App\Configs::get_value -- 167 -- SL_life_time_days does not exist in config (local) !
2019-07-26 15:51:28 -- error -- App\Configs::get_value -- 167 -- landing_page_enable does not exist in config (local) !
2019-07-26 15:50:27 -- error -- App\Configs::get_value -- 167 -- SL_life_time_days does not exist in config (local) !
2019-07-26 15:50:27 -- error -- App\Configs::get_value -- 167 -- landing_page_enable does not exist in config (local) !
Check the Diagnostic page. 🤔
Diagnostics
-----------
Warning: Dropbox import not working. dropbox_key is empty.
System Information
------------------
Lychee-front Version: 3.2.16
Lychee Version (git): 2270caf (master) - Up to date.
DB Version: 040000
System: Linux
PHP Version: 7.3
MySQL Version: 10.3.16-MariaDB-1:10.3.16+maria~stretch
Lychee total space: 4.09 GB
Upload folder space: 4.03 GB
System total space: 13.72 GB
System free space: 9.17 GB (66%)
Imagick: 1
Imagick Active: 1
Imagick Version: 1687
GD Version: 2.2.5
Config Information
------------------
version: 040000
check_for_updates: 1
sorting_Photos_col: takestamp
sorting_Photos_order: ASC
sorting_Albums_col: title
sorting_Albums_order: ASC
imagick: 1
skip_duplicates: 0
small_max_width: 0
small_max_height: 360
medium_max_width: 1920
medium_max_height: 1080
lang: fr
layout: 1
image_overlay: 1
image_overlay_type: exif
default_license: none
compression_quality: 90
full_photo: 1
delete_imported: 1
Mod_Frame: 0
Mod_Frame_refresh: 30000
landing_page_enable: 0
landing_owner: Nicolas
landing_title: Nicolas
landing_subtitle: Cat, Dogs & Humans Photography
landing_facebook: https://www.facebook.com/JohnSmith
landing_flickr: https://www.flickr.com/JohnSmith
landing_twitter: https://www.twitter.com/JohnSmith
landing_instagram: https://instagram.com/JohnSmith
landing_youtube: https://www.youtube.com/JohnSmith
landing_background: dist/cat.jpg
thumb_2x: 1
small_2x: 0
medium_2x: 0
site_title: Lychee v4
site_copyright_enable: 1
site_copyright_begin: 2019
site_copyright_end: 2019
public_search: 0
additional_footer_text:
display_social_in_gallery: 0
public_recent: 0
recent_age: 1
public_starred: 0
downloadable: 0
photos_wraparound: 1
No errors. Then you are fine. 👍
Download is working fine with 500 pictures ;)
Thanks you all ;)