Webp v0.9.3 is consuming a lot of memory again

Question

Webp v0.9.3 is consuming a lot of memory again

Ingiboy opened this issue a year ago · comments

Webp used both for converting to webp and resizing pictures. ~83k requests per day. Щn version 0.8.4 was consistently below 0.5gb

docker-compose.yml

version: "3.3"

services:
  webp:
    container_name: webp
    image: webp:alpine
    build:
      context: webp
      dockerfile: Dockerfile
      args:
        - UID=1001
        - GID=1001
        - REPO=https://github.com/webp-sh/webp_server_go.git
        - TAG=0.9.3
    restart: unless-stopped
    user: 1001:1001
    cap_drop:
      - ALL
    environment:
      - MALLOC_ARENA_MAX=1
      - LD_PRELOAD=/usr/lib/libjemalloc.so.2
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - ./webp/config.json:/etc/config.json:ro
      - /var/tmp/webp:/opt
    ports:
     - 127.0.0.1:3335:3333

Dockerfile:

FROM alpine:3.18 as builder

ARG REPO
ARG TAG

RUN set -ex && \
    apk --update upgrade --no-cache && \
    apk --update add --no-cache git go vips-dev && \
    mkdir /build && \
    cd /build && \
    git clone ${REPO} . && \
    git checkout tags/${TAG} && \
    go mod download && \
    go build -ldflags="-s -w" -o webp-server .


FROM alpine:3.18

ARG UID
ARG GID

RUN set -ex && \
    addgroup -g ${GID} user && adduser -u ${UID} -G user -s /bin/sh -D user && \
    apk --update upgrade --no-cache && \
    apk --update add --no-cache vips ca-certificates jemalloc

COPY --from=builder /build/webp-server  /usr/bin/webp-server
COPY --from=builder /build/config.json /etc/config.json

WORKDIR /opt

CMD ["/usr/bin/webp-server", "--config", "/etc/config.json"]

config.json:

{
  "HOST": "0.0.0.0",
  "PORT": "3333",
  "QUALITY": "80",
  "MAX_JOB_COUNT": "4",
  "IMG_PATH": "http://nginx:8080",
  "EXHAUST_PATH": "./exhaust",
  "ALLOWED_TYPES": ["jpg","png","jpeg","bmp"],
  "ENABLE_EXTRA_PARAMS": true,
  "ENABLE_AVIF": false
}

Nova Kwok · Answer 1 · Mon Jul 17 2023 20:05:23 GMT+0800 (China Standard Time)

Thanks for reporting, comparing 0.8.4 to 0.9.3 0.8.4...0.9.3, there are two possible position that might lead to this problem.

One is GIF support 90680be that changes LoadImageFromFile's param
One is our refactor: 23bbed8

Could you please have a test using our official image and to see if this problem persists? (To mitigate possible alpine problems)
(BTW, what's the main reason for you to build image yourself?

@BennyThink I think we might need a way to help test mem usage on each version, maybe can be triggered by make mem_test, this command should:

Generate lots of images
Send lots of hybrid requests to server
Render RAM usage chart

@Ingiboy How is your RAM usage chart in this issue genereated?

Ingiboy · Answer 2 · Mon Jul 17 2023 20:19:56 GMT+0800 (China Standard Time)

Ram usage chart: docker stats webp --no-stream | grep webp | awk '{print $4}' >>log.txt every 5 min and make diagram in google sheets.
On my server webp converts jpeg and png only, no gif/apng/bmp/etc.

I use alpine for resource saving and safety reasons. As I recall, some of the debian/ubuntu based images did not run with cap_drop: - ALL unlike alpine. I will test your image and report back. Thanks for the quick reply!

Ingiboy · Answer 3 · Mon Jul 17 2023 22:14:24 GMT+0800 (China Standard Time)

The official image exceeded 1gb in less than an hour of work

Nova Kwok · Answer 4 · Mon Jul 17 2023 22:39:50 GMT+0800 (China Standard Time)

@Ingiboy Thanks for you feedback, we're looking at this issue now.

Nova Kwok · Answer 5 · Tue Jul 18 2023 15:45:17 GMT+0800 (China Standard Time)

I've done some testing on some of the versions, and found that this might be the problem that only occurs after 0.8.4, the test procedure as follows:

psrecord for monitoring CPU and RAM usage, example command psrecord 2783701 --plot plot.png
Some sample images totaling of 2.4G
Using find * -type f -print | xargs -I {} echo "curl -H 'User-Agent: Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:98.0) Gecko/20100101 Firefox/98.0' http://localhost:3333/{}" under ./pics for generating test command, this command should output lots of curl command like below:

curl -H 'User-Agent: Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:98.0) Gecko/20100101 Firefox/98.0' http://localhost:3333/DSC05953.jpg
curl -H 'User-Agent: Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:98.0) Gecko/20100101 Firefox/98.0' http://localhost:3333/DSC05954.jpg
curl -H 'User-Agent: Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:98.0) Gecko/20100101 Firefox/98.0' http://localhost:3333/DSC05955.jpg
curl -H 'User-Agent: Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:98.0) Gecko/20100101 Firefox/98.0' http://localhost:3333/DSC05956.jpg
curl -H 'User-Agent: Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:98.0) Gecko/20100101 Firefox/98.0' http://localhost:3333/DSC05957.jpg
curl -H 'User-Agent: Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:98.0) Gecko/20100101 Firefox/98.0' http://localhost:3333/DSC05958.jpg
...

Here are the test results:

0.8.4 with jemalloc

0.9.1 with jemalloc

0.9.3 with jemalloc

0.9.4 with jemalloc

0.9.4 with jemalloc and ReductionEffort set to 0

From the images above we can see after 0.9, the RAM usage is only about 300MB bigger than 0.8.x.(1200MB vs 1600MB)

Also we found out that after 0.9.x, with the introduction of ReductionEffort set to 4 (to solve issue #234), the convertion time has been slowed down to 3 times.

Future steps might be:

Setup tests to cover all versions and to automate tests on RAM and CPU usage to prevent future versions have worse performance
Change ReductionEffort to 0 by default, and if it failed to convert, retry using 4, for better performance
Dig in more on the encode error when ReductionEffort is 0 on some files.

(And there might be a possibility that ReductionEffort set to 4 causes more RAM usage.

Nova Kwok · Answer 6 · Tue Jul 18 2023 16:51:17 GMT+0800 (China Standard Time)

@Ingiboy We've released 0.9.5 to set ReductionEffort to 0 bydefault, would you like to have a test to see if the RAM usage issue mitigated?

Ingiboy · Answer 7 · Tue Jul 18 2023 18:41:42 GMT+0800 (China Standard Time)

@n0vad3v thanks! I'll try to run it on the prod today-tomorrow.

Ingiboy · Answer 8 · Fri Jul 21 2023 21:37:37 GMT+0800 (China Standard Time)

I tested v0.9.4 for 3 days, now run 0.9.7

Ingiboy · Answer 9 · Tue Jul 25 2023 17:47:44 GMT+0800 (China Standard Time)

0.9.8 looks like best version

Nova Kwok · Answer 10 · Tue Jul 25 2023 22:43:46 GMT+0800 (China Standard Time)

@Ingiboy That's great to hear.

In the meantime, we've built a benchmark framework(some how), and done some benchmarks on convert performance and RAM usage from 0.8.0 to 0.9.8.

Benchmark result: https://docs.webp.sh/benchmark/
Benchmark framework: https://github.com/webp-sh/webp_bench

Ingiboy · Answer 11 · Tue Jul 25 2023 23:18:31 GMT+0800 (China Standard Time)

I made a test bench too. But tests and production are very different, the test one is stable and consumes a lot of memory, while the production one has incomprehensible spikes in memory consumption under fairly stable small load.

Nova Kwok · Answer 12 · Wed Aug 09 2023 21:20:35 GMT+0800 (China Standard Time)

But tests and production are very different

Agree, do you have some idea on how to improve theses tests?(Maybe on https://github.com/webp-sh/webp_bench), as theses tests are quite important in my view, many times when we are adding new features, we need to ensure that there are no bug regressions or performance declines.

Ingiboy · Answer 13 · Thu Aug 24 2023 17:59:34 GMT+0800 (China Standard Time)

@n0vad3v, I apologize for such a late reply. About a month ago I accepted a job offer and for the next few months I will hardly be able to take an active part in the development of your very interesting and important project. The previous stack remains in my support but without active development for some time.

I will continue to follow your project anyway, in my opinion, it implements a very necessary functionality. I really hope that you will be able to commercialize it, it deserves it.