Discussion : Image Encoding
shizurumaya opened this issue · comments
Hello,
I noticed that the checkpoint you're using is only 15 MB in size, while the meta checkpoints referenced here (https://github.com/facebookresearch/segment-anything#model-checkpoints) are over 2 GB. Could you please clarify the reason for this discrepancy?
Thank you in advance for your insights.
Hi @shizurumaya . This checkpoint is not from the original SAM. In order to allow users to interact in real time to use SAM to segment objects, we modified the SAM workflow (see https://geo-sam.readthedocs.io/en/latest/#introduction). This checkpoint was created by us to avoid having to encode the image each time new prompts are provided.
Thank you for the response. This tool seems very promising. It's already working well on my end :)
The size of the elements seems to be limited by the tile size. For instance, when I select a large forest, I get a square tile. Is there any plan for the tool to be able to outline elements that are larger than the patch size?
Additionally, I see that the size limit during encoding is 1024. Would the segmented elements potentially be larger if this limit was increased to 2048?
Hi @shizurumaya , you have the option to adjust the resolution while encoding the image. Additionally, we have developed a copilot tool to assist users in determining the appropriate resolution.To access this tool, please download the pre-release version. For more detailed information, please refer to the following link: Encoder Copilot Tool (QGIS).
Currently, SAM only supports image inputs with a resolution of 1024x1024 pixels. As mentioned in my previous comment, reducing the image resolution is a viable solution for most scenarios. However, it seems that changing the resolution may still not cover the entire forest in your specific use case. We are aware of this limitation, which also applies to delineating large glacial areas. We are actively exploring potential solutions to address this issue, but we have not found a satisfactory approach yet. One possible but cumbersome workaround is to manually delineate and save only the boundary portions, then merge these polygons and extract the outer boundary line to achieve the desired extraction. However, this approach poses several challenges and may not provide the best user experience. It is indeed a challenging problem that requires further consideration.
Hi @Fanchengyan, can you check https://github.com/opengeos/segment-geospatial?
I think you might find a solution for 1024x1024 within their code. it's not a QGIS plugin but it works on tif images and might be useful for you.
Hi @4del-Yousefi . Their work also requires resizing the original image. We now have a preliminary idea for solving extremely large objects, and have added it to the Future Works section.