easy installable dictionary

Question

easy installable dictionary

izziiyt opened this issue 5 years ago · comments

Asking PyPI organization allowing 60MB limit exception for full and core dictionary.
This issue is heavily related to https://github.com/WorksApplications/SudachiDict

Yuto Ichikawa commented 5 years ago

@sorami

Paul O'Leary McCann · Answer 1 · Sun Nov 10 2019 13:45:16 GMT+0800 (China Standard Time)

Any progress on this? I think all you need to do is open an issue here and explain why the package is going to be large.

Sorami Hisamoto · Answer 2 · Tue Nov 12 2019 18:34:50 GMT+0800 (China Standard Time)

Hi, thanks for the reminder. Let me have a look and take action in the next few days.

Sorami Hisamoto · Answer 3 · Fri Nov 15 2019 18:58:04 GMT+0800 (China Standard Time)

Okay, so I will confirm with the team about the dictionary file and then open an issue on Pypa repo early next week.

Sorami Hisamoto · Answer 4 · Wed Nov 20 2019 15:08:30 GMT+0800 (China Standard Time)

I've added the following 3 dictionary packages on PyPI;

small (40MB): https://pypi.org/project/SudachiDict-small/20191030/
core (70MB): https://pypi.org/project/SudachiDict-core/0.0.0/
full (150MB): https://pypi.org/project/SudachiDict-full/0.0.0/

The PyPI size limit is 60MB; For small I have uploaded the dictionary resource already, and for core and full I have created the PyPI package version 0.0.0 without the resource files and waiting for the PyPA to increase the size limit.

So you can already do the following to start using the tokenizer;

$ pip install sudachipy sudachidict-small

I have filed an issue to request the size limit increase;
pypa/packaging-problems#299

Once they increase the limit, I will upload core and full resources on PyPI, notify you here, and update the SudachiPy readme.

Hiroshi Matsuda · Answer 5 · Sat Dec 07 2019 11:02:34 GMT+0800 (China Standard Time)

@sorami Did you get some response from the PyPI? I'm going to release the new version of GiNZA in next two weeks. I'd like to make the GiNZA packages available via the PyPI if the SudachDict-core would be also coming from the PyPI.

Sorami Hisamoto · Answer 6 · Sat Dec 07 2019 11:48:33 GMT+0800 (China Standard Time)

@hiroshi-matsuda
Unfortunately, I haven't heard anything from the PyPA team.

The same request made by someone else a day before us ( PyPI package size limit for splice-beakerx · Issue #298 · pypa/packaging-problems ) is in a same situation.

I've added a comment to the issue just now to ping them.

As soon as they change the size limit, our side is ready to release all three (small, core, full) dictionaries on the PyPI.

Paul O'Leary McCann · Answer 7 · Sat Jan 04 2020 15:27:35 GMT+0800 (China Standard Time)

Just poking this issue. It looks like the correct place to make a request for a size increase is actually pypi-support, not the place linked to before. The 298 issue linked here was migrated by the admin there but for some reason the SudachiPy one wasn't, so I guess you need to open a new issue.

Sorami Hisamoto · Answer 8 · Mon Jan 06 2020 15:31:53 GMT+0800 (China Standard Time)

Thank you very much for the information, @polm !!

I have opened an issue there; pypi/support#131

Sorami Hisamoto · Answer 9 · Mon Jan 06 2020 23:05:05 GMT+0800 (China Standard Time)

Limit Request: sudachidict-{core,full} - {75, 160MB} · Issue #131 · pypa/pypi-support

Okay, so we can't distribute the dictionaries on PyPI. Let me consider the alternative approaches Jason introduced to us in the above issue.

Sorami Hisamoto · Answer 10 · Tue May 12 2020 19:28:37 GMT+0800 (China Standard Time)

I've (finally) set up the Python packages for the dictionaries; Now you can install them via PyPI.

$ pip install sudachidict_core
$ pip install sudachidict_small
$ pip install sudachidict_full

The dictionary binary files are not in the packages, but they are downloaded upon installation.