This project focuses on establishing a framework for hyperspectral image classification using deep learning methods. As is well known, supervised deep learning methods are currently state of the art for many machine learning problems, but these methods require large quantities of labeled data to be effective. Unfortunately, existing labeled HSI benchmarks are too small to directly train a deep supervised network. Alternatively, I used self-taught learning, which is an unsupervised method to learn feature extracting frameworks from unlabeled hyperspectral imagery. These models learn how to extract generalizable features by training on sufficiently large quantities of unlabeled data that are distinct from the target data set. Once trained, these models can extract features from smaller labeled target data sets.
In the first stage of the experiment, I used both 2d-convolutional networks and 3d-convolutional networks as autoencoder to produce encoded features. The networks were pre-trained by Hyperion data, and were applied to the Indian Pines data set. Compared with PCA, the results produced by DeHIC is very promising!
Results for the first phase of experiments is shown as Table 2.
The author is working on further experiments, and this repository will be updated as more promising results come out. This introduction will focus on explaining how to process labeled and unlabeled hyperspectral images to make them applicable to current deep learning methods. So users interested in deep learning or hyperspectral remote sensing, can try to find some learning materials on the Internet or contact the author(jingge.xiao@gmail.com).
Unlabeled Hyperion data sets were used to pretrain the networks.
Hyperion is an HSI sensor located on NASA’s EO-1 satellite. Hyperion data contain 242 VNIR/SWIR spectral bands; however, after dropping “Bad Bands” and bands seriously affected by the absorption of water vapor, 175 bands are last. The GSD of each HSI data set is 30.5 m, which is larger than that of labeled data sets used in this experiment.
Figure 1 is some examples of used Hyperion data sets
Patches were constructed from Hyperion data sets(shown in Figure 2) for training of networks. Each patch is a hyperspectral cube with shape 180*8*8(bands*rows*columns). The width and height of the cubes were 8 pixels, and the step is also 8 pixels. So there is no overlap between two adjacent patches.
Band resampling is needed for both unlabeled and labeled data to make them have same spectral bands which is essential of the pre-training process of networks. 180 bands were left after band resampling.
The total number of Hyperion patches used in the experiment is 189466.
Labeled Indian Pines data set was used to evaluate DeHIC.
Indian Pines(shown in Figure 3) is a 200*145*145 data set that was collected over Northwestern Indiana. The original Indian Pines data set has 224 bands, but bands 104–108, 150–163, and 220–224 were removed due to atmospheric absorption or low signalto-noise ratio (SNR). The ground-truth contains 16 classes of different crops and crop mixtures. Indian Pines has a GSD of 20 m.
The way of constructing patches from Indian Pines has two significant differences from that of Hypersion data sets.
- The step is 1 in Indian Pines while equals to width and height of patches in Hyperion data sets. So each patch in Indian Pines represents the pixel(target pixel) which is in the centre of the patch.
- In order to make the patches more representative to their target pixels and to make them rotate invariant, I copyed the value of target pixel to pixels on its right, bottom and bottom right(shown in Figure 4).
The total number of Indian Pines patches used in the experiment is 145*145=21025.
- samples_select_pos.py
Randomly select positions of samples for training and validation
- unlabeled_interp.py
Resample of of spectral values
Hypersion data to pretrain autoencoder networks should have the similiar spectral bands with data sets to be classified.
- unlabeled_scale.py
Normalizing pixel values within each band
Mean and std are calculated using all unlabeled data
- unlabeled_produce_patches.py
Splitting original Hyperion data into patches for subsequent networks training
- labeled_preprocess.py
Processing Indian Pines data set.
Including:
Resampling spectral values
Normalizing pixel values within each band
Constructing patches for subsequent networks training and predicting
- labeled_split_2d.py
Splitting Indian Pines data set into training and validation sets according to selected positions
This file is for 2d-convolutional networks.
- labeled_split_3d.py
Splitting Indian Pines data set into training and validation sets according to selected positions
This file is for 3d-convolutional networks.
- cae_2d_12.py
An example of 2d-convolutional autoencoder with 12 encoded features
The autoencoder network is pretrained by nearly 1.9 million unlabeled hyperion patches. As for the classification of Indian Pines dataset, it is trained again by Indian Pines dataset patches without labels. Then the encoder part is extracted and used to produce encoded features. Finally, features generated from convolutional encoder is fed to SVM classifier.
- cae_3d_12.py
An example of 3d-convolutional autoencoder with 12 encoded features
- pca_split.py
Basic process of Indian Pines data set for PCA-based classification
Including:
Normalizing pixel values within each band
PCA transformation
Splitting data into training and validation sets according to selected positions
- svm_pca.py
Basic implement of PCA classifier
- curves_plot.py
Plotting curves of Hyperspectral data by pixels to faciliate spectral resampling
This not necessary for experiments.
- example_mnist_1.py
An example of using convolutional autoencoder on MNIST dataset.
- example_mnist_2.py
Trains a Stacked What-Where AutoEncoder built on residual blocks on the MNIST dataset.
Although natural language processing, target detection, image classification and speech recognition have achieved great accuracy improvement with the development of machine learning in recent years, and many excellent applications have been brought into people's daily lives. The development of remote sensing images classification has not made an outstanding breakthrough, mainly due to the complexity of data, insufficient labeled samples and limited innovative methods. More work is needed in this field.
DeHIC aims at promoting deep learning methods in hyperspectral image classification. It seeks to build a framework that enables deep learning methods to be more accurate and convenient in this field. In the first phase of the experiment, I tried convolutional autoencoder. However, as for methods, there so many others need to evaluate and modify, such as CNN, RNN, LSTM, DBN, etc. I will try to add some residual blocks and deepen the current autoencoder network to see whether it can achieve better results in the next phase. Frankly, any suggestion or “commits” to DeHIC on GitHub is welcomed!