Unable to upload large pdf files for tram

Question

Unable to upload large pdf files for tram

fishzengyu opened this issue 3 years ago · comments

Hi, I was trying to upload some documents, however, it seems there are space limitations. Am I able to change this value to a larger value?

duttad · Answer 1 · Thu Feb 24 2022 23:50:26 GMT+0800 (China Standard Time)

@fishzengyu What size limit are you seeing? I am unable to upload any file. I am running the application on Docker on Windows 10 Home, and trying to upload the file from elsewhere on the Windows 10 machine.

Emmanuelle Vargas Gonzalez · Answer 2 · Fri Feb 25 2022 21:43:15 GMT+0800 (China Standard Time)

I would echo @duttad comments, the more information you can provide us the better to replicate and fix:

What is your host OS?
Are you downloading the Docker image, building the image locally from the Dockerfile, or running the project directly on the host OS?
How big in filesize: 5MB, 10MB, 50MB? How long is the document: 10, 50, 100 pages?
Are you able to provide any output from TRAM logs?

Emmanuelle Vargas Gonzalez · Answer 3 · Sat Feb 26 2022 09:46:24 GMT+0800 (China Standard Time)

Running from this commit 7c9d875 I cannot replicate the issue on host. Uploaded the documents via the Web UI. They got accepted, the UI displayed Queue for the submission until done. The test document details:

Size	Pages	ML Processing Time
~6MB	1602	26 minutes
~25MB	56	2 minutes

I think we will need more evidence to replicate the issue.

Emmanuelle Vargas Gonzalez · Answer 4 · Tue Mar 01 2022 22:54:18 GMT+0800 (China Standard Time)

@fishzengyu and @duttad, could you try the new published release v1.1.3 to check if the issue still persists?

duttad · Answer 5 · Wed Mar 02 2022 05:22:32 GMT+0800 (China Standard Time)

@emmanvg Sure I plan to give that a try. Do I replace image: ghcr.io/center-for-threat-informed-defense/tram:latest with image: ghcr.io/center-for-threat-informed-defense/tram:v1.1.3 in the docker-compose.yml file?

Emmanuelle Vargas Gonzalez · Answer 6 · Thu Mar 03 2022 21:58:02 GMT+0800 (China Standard Time)

No need to change it, with image: ghcr.io/center-for-threat-informed-defense/tram:latest it will automatically pull the latest version for you.

Wojak394 · Answer 7 · Thu Mar 10 2022 19:59:45 GMT+0800 (China Standard Time)

Hi, I confirm that it's not possible to upload reports (I checked Pdf format) that are bigger than 2MB. I suppose that it's reason of Nginx settings. The error I have is: “client intended to send too large body”. I' ve tried this solution: https://www.jamescoyle.net/how-to/1608-nginx-error-client-intended-to-send-too-large-body , but it is not working! Could you tell how to repair this issue and enlarge maximum size?

I use TRAM build in docker container from the Dockerfile.

Lex1020 · Answer 8 · Wed Mar 30 2022 23:42:52 GMT+0800 (China Standard Time)

I'm having the same issue. Any pdf over 2MB and TRAM won't upload it. It won't even start uploading it.

baga · Answer 9 · Thu Mar 31 2022 16:33:11 GMT+0800 (China Standard Time)

I'm having the same issue. Any pdf over 2MB and TRAM won't upload it. It won't even start uploading it.

me too

Mark E. Haase · Answer 10 · Fri Apr 01 2022 01:48:54 GMT+0800 (China Standard Time)

I noticed the same while testing the docker images this week. Seems to be an nginx issue, because it is not reproduced when using tram runserver.

nginx_1 | 2022/03/30 19:53:29 [error] 33#33: *14 client intended to send too large body: 1400792 bytes, client: 172.18.0.1, server: , request: "POST /upload/ HTTP/1.1", host: "localhost:8000", referrer: "http://localhost:8000/"

I think the solution is to set client_max_body_size in nginx.conf (see this StackOverflow thread). I'll make a PR for this when I get back from vacation, then we'll need to release new Docker images.

cyber-remy · Answer 11 · Tue Apr 05 2022 11:42:46 GMT+0800 (China Standard Time)

Hello, I also experienced this issue.

I rectified it in the containers live as they were running.

There are two problems. The first is as @mehaase explains the nginx server has a default client_max_body_size of 1MB. This can be changed easily. I simply put client_max_body_size 20M; in (the tram-nginx container) /etc/nginx/conf.d/default.conf

In the live container I did this with: sudo docker exec -it [container id] vi /etc/nginx/conf.d/default.conf

However, Django also has a limit on file upload so you need to set that as well. To do this you need to set DATA_UPLOAD_MAX_MEMORY_SIZE and FILE_UPLOAD_MAX_MEMORY_SIZE in /tram/src/tram/settings.py. It needs to be set in bytes so FILE_UPLOAD_MAX_MEMORY_SIZE = 2097152 for 20MB. Unfortunately the tram container does not have a text editor (not even vi) so I modified the settings.py file, putting the two settings just under the database configuration, then used sudo docker cp settings.py [container id]:/tram/src/tram/settings.py. If the file doesnt overwrite you can rename the settings.py in the container with sudo docker exec -it [container id] mv /tram/src/tram/settings.py /tram/src/tram/settings-old.py then copy the file again. After this I rebooted the containers, but not using docker-compose because that would wipe the config.

I'd suggest putting those changes into the published containers.

Remy

nanda-rani · Answer 12 · Tue Apr 26 2022 14:39:36 GMT+0800 (China Standard Time)

Hello, I also experienced this issue.

I rectified it in the containers live as they were running.

There are two problems. The first is as @mehaase explains the nginx server has a default client_max_body_size of 1MB. This can be changed easily. I simply put client_max_body_size 20M; in (the tram-nginx container) /etc/nginx/conf.d/default.conf

In the live container I did this with: sudo docker exec -it [container id] vi /etc/nginx/conf.d/default.conf

However, Django also has a limit on file upload so you need to set that as well. To do this you need to set DATA_UPLOAD_MAX_MEMORY_SIZE and FILE_UPLOAD_MAX_MEMORY_SIZE in /tram/src/tram/settings.py. It needs to be set in bytes so FILE_UPLOAD_MAX_MEMORY_SIZE = 2097152 for 20MB. Unfortunately the tram container does not have a text editor (not even vi) so I modified the settings.py file, putting the two settings just under the database configuration, then used sudo docker cp settings.py [container id]:/tram/src/tram/settings.py. If the file doesnt overwrite you can rename the settings.py in the container with sudo docker exec -it [container id] mv /tram/src/tram/settings.py /tram/src/tram/settings-old.py then copy the file again. After this I rebooted the containers, but not using docker-compose because that would wipe the config.

I'd suggest putting those changes into the published containers.

Remy

Can you let me know where exactly in settings.py we need to change DATA_UPLOAD_MAX_MEMORY_SIZE and FILE_UPLOAD_MAX_MEMORY_SIZE. I'm unable to see any such fields.

cyber-remy · Answer 13 · Thu Apr 28 2022 11:21:00 GMT+0800 (China Standard Time)

Hello

You need to add those fields. Because they are not specified it defaults to a very small allocation. So you add a line for each and set your values.

Mark E. Haase · Answer 14 · Fri May 06 2022 03:48:19 GMT+0800 (China Standard Time)

In my testing, DATA_UPLOAD_MAX_MEMORY_SIZE is not needed because the upload() handler does not access reqeust.body or request.POST (it uses request.FILES). See this doc. Also FILE_UPLOAD_MAX_MEMORY_SIZE is not needed since that affects spill over to disk but does not throw exceptions.

Setting it in nginx allows me to upload a 27MB test PDF. I'll open a PR for this.