PyGrid preparation
AlanAboudib opened this issue · comments
Context
SyferText 0.1.0 is meant to work with PyGrid by default. This means that all pipelines are deployed to PyGrid by default when Language.deploy() is called. And all pipelines are loaded from PyGrid by default when syfertext.load(<language_model_name>) is called.
In previous versions of SyferText, language models had to be hosted and downloaded as separate packages (like in spaCy), prior to using them with SyferText API. This will no more be the case in v0.1.0. All language models that SyferText is able to load, will live on a PyGrid.
For this end, we need some preparation to make this possible. Mainly, creating a permanent experimental PyGrid instance that all SyferText users and contributor will be able to play with. And from which, default pipelines will be available such as spacy_en_core_web_lg that was used for tests in previous versions.
Here is a break down of the necessary sub-issues:
Breakdown
- Create an experimental PyGrid on AWS with a minimum of 3 workers
- (might be unnecessary?)Implement
syfertext.connect()for PyGrid
Where to push your PR?
Any PR referencing this issue should be made to branch syfertext_0.1.0
@AlanAboudib I would like to work on this.
It's yours @Nilanshrajput
Should we close this @Nilanshrajput It is done right?
Let merge the latest pr and i will close this after that.