Error while converting large images
astrowq opened this issue · comments
Describe the bug
The server cannot convert an image of large size (14.9 MB).
To Reproduce
Try to convert this figure with the docker deployment option: https://inspirehep.net/files/3ce7ad44b03d925850182b268e87487e.png.
It returns 502 error code; the server exits and restarts (with the restart: always policy on)
Expected behaviour
It should convert the image and return the converted one.
Screenshots and logs
Webp Server Go is Running on http://0.0.0.0:3333
time="2023-06-17 21:37:22" level=info msg="Remote Addr is https://inspirehep.net/files/3ce7ad44b03d925850182b268e87487e.png, fetching info..." func="[118:main.proxyHandler()]"
2023/06/17 21:37:22 [VIPS.info] vips__open_image_write: opening with O_TMPFILE
2023/06/17 21:37:22 [VIPS.info] vips__open_image_write: O_TMPFILE failed!
2023/06/17 21:37:22 [VIPS.info] vips__open_image_write: simple open
Environment (please complete the following information):
- OS: Linux
- version: 0.8.4
Additional context
The problem is with libvips, apparently. The image is fetched from the remote backend successfuly and can be found in the folder /opt/remote-raw.
I did a simple search but failed to find a solution.
Also note that some other large image files have no problem of being processed, e.g. 6 MB (https://inspirehep.net/files/3afe268881ca91f21167f92921656e48)
Hmm, that's indeed some problem with libvips, I can reproduce this issue locally:
INFO[2023-06-18 06:22:49][118:main.proxyHandler()] Remote Addr is https://inspirehep.net/files/3ce7ad44b03d925850182b268e87487e.png, fetching info...
INFO[2023-06-18 06:22:51][133:main.proxyHandler()] Remote file not found in remote-raw path, fetching...
2023/06/18 06:23:28 [VIPS.info] vips__open_image_write: opening with O_TMPFILE
2023/06/18 06:23:28 [VIPS.info] vips__open_image_write: O_TMPFILE failed!
2023/06/18 06:23:28 [VIPS.info] vips__open_image_write: simple open
WARN[2023-06-18 06:23:34][210:main.webpEncoder()] Can't encode source image: vips2webp: unable to encode
Stack:
goroutine 53 [running]:
runtime/debug.Stack()
/usr/local/go/src/runtime/debug/stack.go:24 +0x65
github.com/davidbyttow/govips/v2/vips.handleVipsError()
/home/nova/go/pkg/mod/github.com/davidbyttow/govips/v2@v2.13.0/vips/error.go:38 +0x57
github.com/davidbyttow/govips/v2/vips.handleSaveBufferError(0xc000086960?)
/home/nova/go/pkg/mod/github.com/davidbyttow/govips/v2@v2.13.0/vips/error.go:31 +0x25
github.com/davidbyttow/govips/v2/vips.vipsSaveToBuffer({0x7f6d900061b0, 0x0, 0x2, 0x0, 0x1, 0x50, 0x0, 0x0, 0x1, 0x0, ...})
/home/nova/go/pkg/mod/github.com/davidbyttow/govips/v2@v2.13.0/vips/foreign.go:465 +0xb7
github.com/davidbyttow/govips/v2/vips.vipsSaveWebPToBuffer(0x7f6d900061b0, {0x1, 0x50, 0x0, 0x0, 0x0, {0x0, 0x0}})
/home/nova/go/pkg/mod/github.com/davidbyttow/govips/v2@v2.13.0/vips/foreign.go:386 +0x1f8
github.com/davidbyttow/govips/v2/vips.(*ImageRef).ExportWebp(0xc00010e0a0, 0x61?)
/home/nova/go/pkg/mod/github.com/davidbyttow/govips/v2@v2.13.0/vips/image.go:924 +0xc5
main.webpEncoder({0xc00007c070, 0x61}, {0xc0005c4d80, 0x53}, 0x50, {0x91b8f0?, 0x0?})
/home/nova/Desktop/webp_server_go/encoder.go:202 +0x29a
main.convertImage({0xc00007c070, 0x61}, {0xc0005c4d80, 0x53}, {0x877eae, 0x4}, {0x0?, 0x0?})
/home/nova/Desktop/webp_server_go/encoder.go:98 +0x2db
main.convertFilter.func2()
/home/nova/Desktop/webp_server_go/encoder.go:55 +0x5c
created by main.convertFilter
/home/nova/Desktop/webp_server_go/encoder.go:54 +0x215
to WebP
ERRO[2023-06-18 06:23:34][57:main.convertFilter.func2()] vips2webp: unable to encode
Stack:
goroutine 53 [running]:
runtime/debug.Stack()
/usr/local/go/src/runtime/debug/stack.go:24 +0x65
github.com/davidbyttow/govips/v2/vips.handleVipsError()
/home/nova/go/pkg/mod/github.com/davidbyttow/govips/v2@v2.13.0/vips/error.go:38 +0x57
github.com/davidbyttow/govips/v2/vips.handleSaveBufferError(0xc000086960?)
/home/nova/go/pkg/mod/github.com/davidbyttow/govips/v2@v2.13.0/vips/error.go:31 +0x25
github.com/davidbyttow/govips/v2/vips.vipsSaveToBuffer({0x7f6d900061b0, 0x0, 0x2, 0x0, 0x1, 0x50, 0x0, 0x0, 0x1, 0x0, ...})
/home/nova/go/pkg/mod/github.com/davidbyttow/govips/v2@v2.13.0/vips/foreign.go:465 +0xb7
github.com/davidbyttow/govips/v2/vips.vipsSaveWebPToBuffer(0x7f6d900061b0, {0x1, 0x50, 0x0, 0x0, 0x0, {0x0, 0x0}})
/home/nova/go/pkg/mod/github.com/davidbyttow/govips/v2@v2.13.0/vips/foreign.go:386 +0x1f8
github.com/davidbyttow/govips/v2/vips.(*ImageRef).ExportWebp(0xc00010e0a0, 0x61?)
/home/nova/go/pkg/mod/github.com/davidbyttow/govips/v2@v2.13.0/vips/image.go:924 +0xc5
main.webpEncoder({0xc00007c070, 0x61}, {0xc0005c4d80, 0x53}, 0x50, {0x91b8f0?, 0x0?})
/home/nova/Desktop/webp_server_go/encoder.go:202 +0x29a
main.convertImage({0xc00007c070, 0x61}, {0xc0005c4d80, 0x53}, {0x877eae, 0x4}, {0x0?, 0x0?})
/home/nova/Desktop/webp_server_go/encoder.go:98 +0x2db
main.convertFilter.func2()
/home/nova/Desktop/webp_server_go/encoder.go:55 +0x5c
created by main.convertFilter
/home/nova/Desktop/webp_server_go/encoder.go:54 +0x215
By configuring ReductionEffort we can resolve this. The root cause should be your kernel/OS doesn't support O_TMPFILE.
level 2: 5.9s, 7.6MB
level 4: 26s 6.9MB
This will be fixed in the coming version.
Hi, I have tried the version 0.9.0. Unfortunately, the same error still exists for the image file with size of 14.9 MB.
I am using the official docker image. Any idea on how to resolve the point you mentioned that kernel/OS doesn't support O_TMPFILE?
Many thanks.
ohh ReductionEffort should be set to 4(default value) for better compatibility.
@astrowq
We've released 0.9.1 that could fix this problem, I've tested locally(using docker container) and it's working now
.
It is working on my server too, after upgrading to 0.9.1. Thank you very much for the quick fix!