Unsetting Model State
erosenthal-square opened this issue · comments
Is it possible to unset a model state? I'm interested in changing a model from a shadow state to a production state such that the model no longer shows up if I query
model_store.list_versions("my-domain", state_name="shadow")I see that I can set a model state, and that a model can be set to multiple different states at the same time, but I don't see any support for removing a model from a given state.
Hi @erosenthal-square! Thanks for raising this.
A couple thoughts on my side:
-
I think you've actually found a bug here; I had originally intended for model:states to be 1:1 rather than 1:many. So when a model is changed from
"shadow"to"production"it should only be set to"production"and not appear when listing"shadow"models. -
Bug aside, I think this feature would still be useful to have, to enable folks to set a
"shadow"model to have no state. I can see this working in one of two ways (below) - which one strikes you as more intuitive?
# Unset a state by setting a state to None
model_store.set_model_state("my-domain", "model-id", state_name=None)
# Unset a state by removing that particular state
model_store.remove_model_state("my-domain", "model-id", state_name="shadow")
I can investigate this further on Friday!
Hey, thanks for the great library!
Yeah, I had originally assumed that it was 1:1 model:states prior to running into this. If you want to maintain the current behavior of 1:many, then I think your second example feels more intuitive. For the first example, it's unclear to me what would happen if I ran that code while my model was in two different states.
If you want to change the logic to 1:1, then I think the first example is a more intuitive way to "unset" the state.
Cool, that makes sense! Will think about this a bit 🤔 - it's not a trivial fix when thinking about folks who are already using modelstore!
In the mean time, there is a manual way that you can "unset" the state of a model which can unblock you in the short term.
Under the hood, setting a state creates a file at a pre-defined path (defined by this function). Then, when modelstore lists models that have a given state, it is just reading the files that exist at that path (here). So if you manually delete that file, the state will be unset.
The way to do this is to navigate to whichever storage layer you're using (s3 bucket, file system), and you'll find the file in:
operatorai-model-store/<domain-name>/versions/<state-name>/<model-id>.json
And if you remove that file from that path, then the model state will be unset 👍
This is just a short-term solution until I get a proper fix out!
Cool, I'm happy to hack the manual solution in the meantime. Thanks!
@erosenthal-square This is now merged into the main branch ✅ -- it'll go out the next time I release to pip too, which should be soon (I'll leave this issue open until then). I've gone with the quicker of the two fixes - which adds:
model_store.remove_model_state(model_domain, model_id, state_name)
And preserves the current (many:many) mapping between models and states. Will look to make that clearer in the future 😄 Thanks for raising this, I'd appreciate any other ideas or feedback you have 🙏
Thanks so much!
This is shipped and now available on pypi in modelstore==0.0.72 ✅