Unable to upload large pdf files for tram
fishzengyu opened this issue · comments
Hi, I was trying to upload some documents, however, it seems there are space limitations. Am I able to change this value to a larger value?
@fishzengyu What size limit are you seeing? I am unable to upload any file. I am running the application on Docker on Windows 10 Home, and trying to upload the file from elsewhere on the Windows 10 machine.
I would echo @duttad comments, the more information you can provide us the better to replicate and fix:
- What is your host OS?
- Are you downloading the Docker image, building the image locally from the Dockerfile, or running the project directly on the host OS?
- How big in filesize: 5MB, 10MB, 50MB? How long is the document: 10, 50, 100 pages?
- Are you able to provide any output from TRAM logs?
Running from this commit 7c9d875 I cannot replicate the issue on host. Uploaded the documents via the Web UI. They got accepted, the UI displayed Queue
for the submission until done. The test document details:
Size | Pages | ML Processing Time |
---|---|---|
~6MB | 1602 | 26 minutes |
~25MB | 56 | 2 minutes |
I think we will need more evidence to replicate the issue.
@fishzengyu and @duttad, could you try the new published release v1.1.3
to check if the issue still persists?
@emmanvg Sure I plan to give that a try. Do I replace image: ghcr.io/center-for-threat-informed-defense/tram:latest
with image: ghcr.io/center-for-threat-informed-defense/tram:v1.1.3
in the docker-compose.yml file?
No need to change it, with image: ghcr.io/center-for-threat-informed-defense/tram:latest
it will automatically pull the latest version for you.
Hi, I confirm that it's not possible to upload reports (I checked Pdf format) that are bigger than 2MB. I suppose that it's reason of Nginx settings. The error I have is: “client intended to send too large body”. I' ve tried this solution: https://www.jamescoyle.net/how-to/1608-nginx-error-client-intended-to-send-too-large-body , but it is not working! Could you tell how to repair this issue and enlarge maximum size?
I use TRAM build in docker container from the Dockerfile.
I'm having the same issue. Any pdf over 2MB and TRAM won't upload it. It won't even start uploading it.
I'm having the same issue. Any pdf over 2MB and TRAM won't upload it. It won't even start uploading it.
me too
I noticed the same while testing the docker images this week. Seems to be an nginx issue, because it is not reproduced when using tram runserver
.
nginx_1 | 2022/03/30 19:53:29 [error] 33#33: *14 client intended to send too large body: 1400792 bytes, client: 172.18.0.1, server: , request: "POST /upload/ HTTP/1.1", host: "localhost:8000", referrer: "http://localhost:8000/"
I think the solution is to set client_max_body_size
in nginx.conf (see this StackOverflow thread). I'll make a PR for this when I get back from vacation, then we'll need to release new Docker images.
Hello, I also experienced this issue.
I rectified it in the containers live as they were running.
There are two problems. The first is as @mehaase explains the nginx server has a default client_max_body_size of 1MB. This can be changed easily. I simply put client_max_body_size 20M;
in (the tram-nginx container) /etc/nginx/conf.d/default.conf
In the live container I did this with: sudo docker exec -it [container id] vi /etc/nginx/conf.d/default.conf
However, Django also has a limit on file upload so you need to set that as well. To do this you need to set DATA_UPLOAD_MAX_MEMORY_SIZE and FILE_UPLOAD_MAX_MEMORY_SIZE in /tram/src/tram/settings.py. It needs to be set in bytes so FILE_UPLOAD_MAX_MEMORY_SIZE = 2097152
for 20MB. Unfortunately the tram container does not have a text editor (not even vi) so I modified the settings.py file, putting the two settings just under the database configuration, then used sudo docker cp settings.py [container id]:/tram/src/tram/settings.py
. If the file doesnt overwrite you can rename the settings.py in the container with sudo docker exec -it [container id] mv /tram/src/tram/settings.py /tram/src/tram/settings-old.py
then copy the file again. After this I rebooted the containers, but not using docker-compose because that would wipe the config.
I'd suggest putting those changes into the published containers.
Remy
Hello, I also experienced this issue.
I rectified it in the containers live as they were running.
There are two problems. The first is as @mehaase explains the nginx server has a default client_max_body_size of 1MB. This can be changed easily. I simply put
client_max_body_size 20M;
in (the tram-nginx container) /etc/nginx/conf.d/default.confIn the live container I did this with:
sudo docker exec -it [container id] vi /etc/nginx/conf.d/default.conf
However, Django also has a limit on file upload so you need to set that as well. To do this you need to set DATA_UPLOAD_MAX_MEMORY_SIZE and FILE_UPLOAD_MAX_MEMORY_SIZE in /tram/src/tram/settings.py. It needs to be set in bytes so
FILE_UPLOAD_MAX_MEMORY_SIZE = 2097152
for 20MB. Unfortunately the tram container does not have a text editor (not even vi) so I modified the settings.py file, putting the two settings just under the database configuration, then usedsudo docker cp settings.py [container id]:/tram/src/tram/settings.py
. If the file doesnt overwrite you can rename the settings.py in the container withsudo docker exec -it [container id] mv /tram/src/tram/settings.py /tram/src/tram/settings-old.py
then copy the file again. After this I rebooted the containers, but not using docker-compose because that would wipe the config.I'd suggest putting those changes into the published containers.
Remy
Can you let me know where exactly in settings.py we need to change DATA_UPLOAD_MAX_MEMORY_SIZE and FILE_UPLOAD_MAX_MEMORY_SIZE. I'm unable to see any such fields.
Hello
You need to add those fields. Because they are not specified it defaults to a very small allocation. So you add a line for each and set your values.
In my testing, DATA_UPLOAD_MAX_MEMORY_SIZE
is not needed because the upload()
handler does not access reqeust.body
or request.POST
(it uses request.FILES
). See this doc. Also FILE_UPLOAD_MAX_MEMORY_SIZE
is not needed since that affects spill over to disk but does not throw exceptions.
Setting it in nginx allows me to upload a 27MB test PDF. I'll open a PR for this.