orchest / orchest

Build data pipelines, the easy way 🛠️

Home Page:https://orchest.readthedocs.io/en/stable/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

import bigger file to project directory

nuhyurdev opened this issue · comments

Hello,

I imported csv file to orchest project directory but a huge file (> 500MB ) is not imported. I didn't get any errors. This feature needs an improvement.
And Is there any way to do this ? I need to import huge files to my projects.

We'll investigate limits here, it shouldn't "inherently" be impossible but you might be hitting certain limits we've overlooked (we're not capping the upload size ourselves).

An alternative you can use in the meanwhile is using Python or the JupyterLab terminal to download a file from a URL using something like wget/curl or a Python package like requests.

E.g. run cd /data && wget https://<somefile> in the JupyterLab environment shell terminal (https://docs.orchest.io/en/stable/fundamentals/environments.html#environment-shells).

Thanks for reporting this issue 🙏

Hi @nuhyurduseven, it looks like the max body size is not being correctly set in some cases, as a temporary workaround you could patch the ingress object through this command:

kubectl patch -n orchest ingress orchest-webserver -p '{"metadata":{"annotations":{"nginx.ingress.kubernetes.io/proxy-body-size":"0"}} }'

Unfortunately, this needs to be reapplied when restarting or updating Orchest. The solution to this issue should be fairly simple so we'll likely release a fix pretty soon.

Hi @fruttasecca, Thanks for the solution. I fetched the data from database in orchest yesterday Now I met my need. But I'll try your solution or I think I can wait your update.

@nuhyurduseven happy to hear that. I'll keep the GH issue open in order to keep track of it.

@nuhyurduseven : Thanks for reporting this, this issue has been addressed in #1362 , and will be fixed in the next release.

Thanks. I like the tool and I needed something just like this. Because of that, I want to help improve the tools and become a tester for you. You can be sure that I will push the limits of the tool in the future.