Geo SAM

By Joey and Fancy from Cryosphere Lab, ESSC, CUHK.

Introduction

Geo SAM is a QGIS plugin tool that aims to help people segment, delineate or label landforms efficiently when using large-size geospatial raster images. Segment Anything Model (SAM) is a foundation AI model with the superpower, but the model size is huge, and using it to process images can take a long time, even with a modern GPU. Our tool uses the strategies of encoding image features in advance and trimming the SAM model. The interactive segmentation process can be run in real-time on a laptop by only using a CPU, making it a convenient and efficient tool for dealing with satellite images.

The tool currently only supports preprocessed images (whose features have been generated in advance using a separate program, as the included demo image). We are now building another tool for encoding image features inside QGIS, which will soon be available. So stay tuned.

Installation

Install QGIS

You are suggested to install the latest version of QGIS since the plugin has only been tested on the versions newer than QGIS 3.30 (at least ver. 3.28 is recommended).

Install Library Dependencies

For Windows Users

Open the OSGeo4W Shell application from the Start menu, which is a dedicated shell for the QGIS. Then run the following command to install the libraries.

pip3 install torch==1.13.1 torchvision==0.14.1
pip3 install torchgeo
pip3 install segment-anything
pip3 install rasterio==1.3.7

For Mac or Linux Users

Open your own terminal application, and change the directory to the QGIS Python environment.

# Mac
cd /Applications/QGIS.app/Contents/MacOS/bin
# Linux (Please confirm the python env by "import qgis")
cd /usr/bin

To confirm the QGIS Python environment:

./python3

>>> import qgis

Then install the libraries.

# add ./ to avoid using your default Python in the system
./pip3 install torch==1.13.1 torchvision==0.14.1
./pip3 install torchgeo
./pip3 install segment-anything
./pip3 install rasterio==1.3.7

For Linux users, if pip3 is not found in /usr/bin, try the following commands:

sudo apt-get update
sudo apt-get install python3-pip

Install the GeoSAM Plugin

Download the plugin zip file, unzip it, and put the Geo-SAM folder (please remove the version suffix of the folder to avoid potential path issues, be aware of undesired nested folders after unzipping) into the QGIS plugin folder, then restart QGIS if it's open already.

How to Locate the QGIS Plugin folder

From the Settings Menu, select User Profiles, then select Open active profile folder. You'll be taken straight to the profile directory in Explorer or Finder. Under the profile folder, you may find a python folder; the plugins folder should be right inside the python folder. Open the plugins folder, then put the entire Geo-SAM folder in it, then restart QGIS.

Below are some general paths of different systems for your reference.

# Windows
%APPDATA%\QGIS\QGIS3\profiles\default\python\plugins
# Mac
~/Library/Application\ Support/QGIS/QGIS3/profiles/default/python/plugins
# Linux
~/.local/share/QGIS/QGIS3/profiles/default/python/plugins

Activate Geo SAM Plugin

After restarting QGIS, you may go to the Plugins menu, select Manage and Install Plugins, and under Installed, you may find the Geo SAM plugin; check it to activate the plugin.

Find the Geo SAM Tool

After activating the Geo SAM plugin, you may find the tool under the Plugins menu,

Or somewhere on the toolbar near the Python Plugin.

Use the GeoSAM Tool

Click the toolbar icon to open the widget of the tool. You will be shown a demo raster image with thaw slump and small pond landforms for you to try the tool. With a single click on the map, a segmentation result will be generated.

A user interface will be shown below.

Add Points

Click the buttons to select between the Foreground and Background points. Use Foreground points to add areas you desire, and use Background points to remove areas you don't want.

Add Bounding Box (BBox)

Click the Rectangle button to activate the BBox tool to draw a rectangle on the map for segmenting a subject. The BBox tool can be used together with adding points or independently.

Save Current Results

After adding points and a rectangle for segmenting a subject, you can save the segmentation results by clicking the Save button.

Clear Points and BBox

You can use the Clear button to clear the added points and rectangles.

Undo the last Prompt

You can use the Undo button to undo the last points or rectangle Prompt.

Disable the Tool

You can uncheck the Enable button to temporally disable the tool and navigate on the map.

Load Selected Image Features

The plugin is initialized with features for demo purposes, and you can use the Feature Folder selection tool to select the folder that includes the image features you need.

After selecting the feature folder, you may press the Load button to load the features, and it may take several seconds when you load the folder for the first time. Remember to add the corresponding raster image to the QGIS project.

Shortcuts

Tab: loop between 3 prompt types (the cursor will also change to the corresponding types):
- Foreground Point
- Background Point
- Rectangle/BBox
C: clear all prompts in canvas [same as Clear button]
Z: undo the last prompt in canvas [same as Undo button]
S: save SAM output features into polygon [same as Save button]
Ctrl+Z or command+Z: undo the last saved features of SAM output

Tips for Using GeoSAM Tool

Deal with only One Subject each time
Use Background Points to exclude unwanted parts
Use Bounding Box (BBox) to limit the segment polygon boundary
The BBox should cover the entire subject
Remember to press the Save button after the segmentation of the chosen subject

In Progress

Image encoder module

Future Works

Existing polygon refinement

Acknowledgement

This repo benefits from Segment Anything and TorchGeo. Thanks for their wonderful work.

cryoyan / Geo-SAM