ksugar / qupath-extension-sam

Hi @ksugar,
we would like to implement loading of custom SAM models (e.g. from micro_sam) in the QuPath plugin.
This should be easy on the server side, and I think my PR in ksugar/samapi#11 already adds everything that's necessary there.

But I am not so sure what needs to be changed here. From a quick check I think I would need to add the option for a custom URL here:

qupath-extension-sam/src/main/java/org/elephant/sam/SAMCommand.java

Lines 101 to 102 in 609289d

    
           private final ObjectProperty<SAMModel> model = PathPrefs.createPersistentPreference( 
        
                   "ext.SAM.model", SAMModel.VIT_L, SAMModel.class);

Is that correct, or is there some other relevant place in the code? Also, how should the default (no custom URL) be represented? In python it's just None.

Thanks!

That sounds great! It looks like the use of an enum for SAMModel would need to change. The way we did it in the WSInfer extension is to use a String or alternative non-enum class instead, and populate the combobox by querying the server for the list of available models on the server.

Current SAM approach here
Alternative WSInfer approach here

Then hopefully adding more models becomes easy, and doesn't require any change to the extension.

Hi @constantinpape, thank you for the PR!
I checked your PR and found that it is a good idea to handle all models (incl. currently available models) with a single interface with just the checkpoint_url argument.

On the client side (i.e., QuPath), in addition to the preset model options (vit_h, vit_l, vit_b and vit_m), the user will have the option to add a custom model by specifying the URL with a nickname. Once it is added, it will remain until the user explicitly deletes it. QuPath's PathPrefs will manage a list of custom urls.

I will work on it later today.

QuPath's PathPrefs will manage a list of custom urls.

I'm not sure PathPrefs will be very good at that by default. The trouble is that Java's Preferences class doesn't support arrays/lists. It should still work if you don't mind squeezing everything into a single string, although this won't support extremely long lists (the maximum string length is 8192 characters).

My guess is that the client supporting a list of models returned by the server would be easier to develop and manage, so that any extra URLs are handled when starting up the server.

Hi @petebankhead, thank you for the feedback.

I was expecting that PathPrefs.createPersistentJsonPreference() can handle arrays/lists of String but it seems that it is not the case.

Following your suggestion, I will implement an endpoint for getting a list of the available models on the server side.

I forgot about PathPrefs.createPersistentJsonPreference() :)
Probably it should work (although I guess maybe doesn't).

Thanks for the quick replies @ksugar and @petebankhead! From what I understand you will take care of this for now @ksugar :). Let me know if you need any more info, or once it's ready to be tested!

Hi @constantinpape and @petebankhead, I have implemented the function for loading custom model weights from URL.

Please try qupath-extension-sam-0.4.0-SNAPSHOT.jar with the updated version of samapi from the dev branch.

python -m pip install -U git+https://github.com/ksugar/samapi.git@dev

Since this version, it is important to launch the server with --workers 2 (or more) to enable cancellation of a download of a weights file.

export PYTORCH_ENABLE_MPS_FALLBACK=1 # Required for running on Apple silicon
uvicorn samapi.main:app --workers 2

The relevant documentation is found here.
https://github.com/ksugar/qupath-extension-sam/tree/dev#register-sam-weights-from-url

Thanks @ksugar ! I will try it out as soon as possible and let you know what I find.

I tested it now @ksugar and it all works as expected!

Thank you @constantinpape for testing it! I will release it soon.
I'm thinking that it would be helpful to have a catalog for weights in README so that the users can find them easily.
How do you think?

Type	Name (customizable)	URL	Citation
vit_h	vit_h_lm	https://zenodo.org/record/8250299/files/vit_h_lm.pth?download=1	Archit, A. et al. Segment Anything for Microscopy. bioRxiv 2023. doi:10.1101/2023.08.21.554208 https://github.com/computational-cell-analytics/micro-sam
vit_b	vit_b_lm	https://zenodo.org/record/8250281/files/vit_b_lm.pth?download=1
vit_h	vit_h_em	https://zenodo.org/record/8250291/files/vit_h_em.pth?download=1
vit_b	vit_b_em	https://zenodo.org/record/8250260/files/vit_b_em.pth?download=1

Thank you @constantinpape for testing it! I will release it soon. I'm thinking that it would be helpful to have a catalog for weights in README so that the users can find them easily. How do you think?
Type Name (customizable) URL Citation
vit_h vit_h_lm https://zenodo.org/record/8250299/files/vit_h_lm.pth?download=1 Archit, A. et al. Segment Anything for Microscopy. bioRxiv 2023. doi:10.1101/2023.08.21.554208

https://github.com/computational-cell-analytics/micro-sam
vit_b vit_b_lm https://zenodo.org/record/8250281/files/vit_b_lm.pth?download=1
vit_h vit_h_em https://zenodo.org/record/8250291/files/vit_h_em.pth?download=1
vit_b vit_b_em https://zenodo.org/record/8250260/files/vit_b_em.pth?download=1

Yes, that would be great!

	private final ObjectProperty<SAMModel> model = PathPrefs.createPersistentPreference(
	"ext.SAM.model", SAMModel.VIT_L, SAMModel.class);

Enable loading custom models in the QuPath extension