Geo SAM

By Joey and Fancy from Cryosphere Lab, ESSC, CUHK.

Geo SAM

Introduction

Geo SAM is a QGIS plugin that aims to help people segment, delineate or label landforms efficiently when using large-size geospatial raster images. Segment Anything Model (SAM) is a foundation AI model with the superpower, but the model size is huge, and using it to process images can take a long time, even with a modern GPU. Our tool uses the strategies of encoding image features in advance and trimming the SAM model. The interactive segmentation process can be run in real-time on a laptop by only using a CPU, making it a convenient and efficient tool for dealing with satellite images.

The Geo SAM plugin includes two separate tools, the encoding tool, and the segmentation tool. The encoding tool is designed to generate and save the image features using the SAM image encoder, and the encoding process only needs to run once per image. The segmentation tool is for interactively segmenting landforms, and it can only be used to segment preprocessed images (whose features have been generated in advance using the encoding tool, as the included demo image).

Installation

Install QGIS

You are suggested to install the latest version of QGIS since the plugin has only been tested on versions newer than QGIS 3.30.

Install Library Dependencies

For Windows Users

Open the OSGeo4W Shell application from the Start menu, which is a dedicated shell for the QGIS. Then run the following command to install the libraries.

pip3 install torch torchvision
pip3 install torchgeo
pip3 install segment-anything

Our encoder tool now supports using CUDA GPU to accelerate the encoding process. If your PC has dedicated CUDA GPUs, you can install the CUDA library first and then install the gpu-version pytorch using the following command (using CUDA version 11.7 as an example):

# add `--force-reinstall` if you installed the cpu version before.
pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu117

For Mac or Linux Users

Open your own terminal application, and change the directory to where the QGIS Python binary file locates.

# Mac
cd /Applications/QGIS.app/Contents/MacOS/bin
# Linux (Please confirm the python env by "import qgis")
cd /usr/bin

To confirm the QGIS Python environment:

./python3
>>> import qgis

Then install the libraries.

# !important, add ./ to avoid using your default Python in the system
./pip3 install torch torchvision
./pip3 install torchgeo
./pip3 install segment-anything

For Linux users, if pip3 is not found in /usr/bin, try the following commands:

sudo apt-get update
sudo apt-get install python3-pip

For Linux users, if your computer got available CUDA GPUs and with CUDA library installed, the above commands should have helped you install the gpu-version pytorch. You can reach pytorch official website for more information.

Install the Geo SAM Plugin

Download the Plugin

Download the plugin zip file, unzip it, and rename the folder as Geo-SAM (be aware of undesired nested folders after unzipping).

Locate the QGIS Plugin folder

In QGIS, Go to the menu Settings > User Profiles > Open active profile folder. You'll be taken straight to the profile directory. Under the profile folder, you may find a python folder; the plugins folder should be right inside the python folder (create the plugins folder if it does not exist). Put the entire Geo-SAM folder inside the plugins folder, then restart QGIS. The directory tree structure should be the same as the following.

python
└── plugins
    └── Geo-SAM
        ├── checkpoint
        ├── docs
        ├── ...
        ├── tools
        └── ui

Below are some general paths of the plugin folder for your reference.

# Windows
%APPDATA%\QGIS\QGIS3\profiles\default\python\plugins
# Mac
~/Library/Application\ Support/QGIS/QGIS3/profiles/default/python/plugins
# Linux
~/.local/share/QGIS/QGIS3/profiles/default/python/plugins

Activate the Geo SAM Plugin

After restarting QGIS, go to the menu Plugins > Manage and Install Plugins, and under Installed, you may find the Geo SAM plugin; check it to activate the plugin.

After activating the Geo SAM plugin, you may find the Geo SAM tools under the Plugins menu,

You may also find a new toolbar, including two icons.

Use the Geo SAM Segmentation Tool

Click the Segmentation Tool icon to open the interactive segmentation widget. You will be shown a demo raster image with thaw slump and small pond landforms for you to try the tool. With a single click on the map, a segmentation result will be generated.

A user interface will be shown below.

Add Prompts

Click the buttons to select between the Foreground(FG) and Background(BG) points. Use FG points to add areas you desire, and use BG points to remove areas you don't want.

Click the BBox button to choose to add a bounding box (BBox) on the canvas by pressing and dragging the mouse. BBox can be used together with adding points or independently.

Save Current Results

After adding points and a BBox for segmenting a subject, you can save the segmentation results by clicking the Save button.

Undo/Clear Prompts

You can use the Undo button to undo the last point or BBox prompt.

You can use the Clear button to clear the added points and BBox.

Enable/Disable the Tool

You can uncheck the Enable button to temporally disable the tool and navigate on the map.

Load Image Features

The plugin is initialized with features for demo purposes, and you can use the Feature Folder selection button to select the folder that includes the image features you need.

Then, press the Load button to load the selected image features. Remember to add the corresponding raster image to the QGIS project.

Shortcuts

Tab: loop between 3 prompt types (the cursor will also change to the corresponding types)
C: clear all prompts in canvas [same as Clear button]
Z: undo the last prompt in canvas [same as Undo button]
S: save SAM output features into polygon [same as Save button]
Ctrl+Z or command+Z: undo the last saved segmentation results

Tips for Using the Segmentation Tool

Deal with only One Subject each time
Use Background Points to exclude unwanted parts
Use Bounding Box (BBox) to limit the segment polygon boundary
The BBox should cover the entire subject
Remember to press the Save button after the segmentation of the chosen subject

Use the Geo SAM Encoding Tool

If you want to try your own images, you can use the Encoding Tool. This tool helps to preprocess geospatial images and generate image features using the SAM image encoder. The generated image features can then be used in our Geo-SAM tool to label the landforms by adding points and bounding box prompts.

Download SAM Checkpoints

SAM model checkpoints should be downloaded in advance, and three versions (huge, large, and base) are available. The large version "vit_l" is recommended to try first. You need to specify the model type that matches the checkpoint version. Using the following links to download the checkpoints.

Select Bands and Value Range for Processing

After selecting the raster layer or image file you want to process, you should also choose the proper bands. The SAM natively supports only three-band RGB images, but we have adapted the tool to support one or two-band images so that you can try grayscale images or NDVI spectral index images.

The values of the image input to the SAM should range from 0 to 255, and you may need to specify the value range (in Advanced Parameters) to be rescaled to [0, 255]. By default, the tool will help you to find the min and max values of the image data and rescale the value range of [min, max] to [0, 255].

Patch Sampling

Since SAM only supports input images with sizes of (1024, 1204), small images will be resized to match the input size, while large images will be sampled into overlapped patches (patch_size=1024) in a grid-like fashion. The stride parameter will determine the overlap behavior, overlap = patch_size - stride.

Demo Animation

The following animation shows how to use the encoder tool.

After processing the image, by default, the generated features will automatically be loaded in the segmentation tool for you to start labeling. Or you can choose to load the image features manually afterward.

Tips for Making the Encoding Process Faster

Choose a smaller processing extent
Reduce target resolution (in Advanced Parameters)
Increase stride to minimize overlap
Choose a smaller version of SAM model
Use GPU
Increase batch_size when using a GPU with sufficient GPU memory

Future Works

Existing polygon refinement

Citation

Zhao, Zhuoyi, Fan, Chengyan, & Liu, Lin. (2023). Geo SAM: A QGIS plugin using Segment Anything Model (SAM) to accelerate geospatial image segmentation (1.1.0). Zenodo. https://doi.org/10.5281/zenodo.8191039

@software{zhao_zhuoyi_2023_8191039,
  author       = {Zhao, Zhuoyi and Fan, Chengyan and Liu, Lin},
  title        = {{Geo SAM: A QGIS plugin using Segment Anything Model (SAM) to accelerate geospatial image segmentation}},
  month        = jul,
  year         = 2023,
  publisher    = {Zenodo},
  version      = {1.1.0},
  doi          = {10.5281/zenodo.8191039},
  url          = {https://doi.org/10.5281/zenodo.8191039}
}

Acknowledgement

This repo benefits from Segment Anything and TorchGeo. Thanks for their wonderful work.

About

A QGIS plugin tool using Segment Anything Model (SAM) to accelerate segmenting or delineating landforms in geospatial raster images.

MIT License

Languages

Language:Python 89.8%Language:Jupyter Notebook 10.2%