An NAENH-ISOMAP (Noise-Aware ENHanced Isometric-Mapping, a high-performance nonlinear dimensionality reduction (manifold learning) algorithm for highly-noisy, large-scale, high-dimensional data) implementation for hyperspectral imagery, and general-purpose dimensionality reduction library / toolset.
hsisomap is now pre-released.
Table of Contents
The initial goal of the project is to demonstrate a novel noise-aware ISOMAP algorithm and workflow for hyperspectral imagery (See references). The core algorithms consisting the workflow have been implemented as a C++ library, and the workflow has been implemented in a commandline tool, as well as a GUI tool to facilitate the use of the workflow in real-world hyperspectral imagery analysis.
At the same time, the project can be used as a general-purpose dimensionality reduction library, as many of the consisting components include commonly used functionality in image analysis and machine learning, such as MNF (Minimum Noise Fraction), PCA (Principal Component Analysis), Landmark ISOMAP, and modules that can be useful such as kNN graph generation, OpenCL based Dijkstra algorithm etc.
These general-purpose modules can be called as static library or used as source code. The commandline and GUI tool can also be used for general-purpose dimensionality reduction for data not limiting to hyperspectral imagery. The commandline tool implements a JSON based human-readable configuration. The GUI tool can be used to facilitate the configuration file generation and task execution.
This section contains a brief description of the components of the project, and the modules of the hsisomap library, as well as some design goals.
The project consists the following build targets:
libhsisomap: static library of hsisomap.hsisomap_cli: commandline tool to execute NAENH-ISOMAP and other tasks with configuration files. Dependent onlibhsisomap.demo: code containing examples to uselibhsisomapdirectly. Dependent onlibhsisomap.hsisomap_gui: GUI tool to facilitate the use ofhsisomap_cliand creating / editing configuration files. Requireshsisomap_cli.gtest: unit-tests forlibhsisomap(dependent).
See Documentation.
hsisomap does not provide pre-compiled releases at this stage. The library and tools should be compiled from source repository.
hsisomap is cross-platformed. It is being tested under macOS and Linux. It should also work under Windows with proper configuration but it has not been tested.
The dependencies to compile libhsisomap, hsisomap_cli and demo includes (the version number followed is the developing / testing environment; early versions may work but not tested):
- GNU Scientific Library (GSL)
2.2.1 - Boost
1.57.0 - OpenCL
2.0
The dependency to compile hsisomap_gui is:
- Qt 5
5.7.1
The compilers tested are:
- Apple LLVM version 8.0.0 (clang-800.0.42.1)
- G++ (TBD)
To compile the project automatically, CMake is also required.
- CMake
3.6.2
Optionally, the Documentation can be generated by Doxygen:
- Doxygen
1.8.13
The commandline tool and GUI tool also use a 3rd-party header file PicoJSON 1.3.1-dev. The file is included in the repository and does not need to be installed separately.
Below are platform-specific instructions to compile the project.
The compiler and OpenCL.framework are automatically installed by Xcode, which can be obtained from the Mac App Store.
The easiest way to install dependencies on macOS is homebrew. The link contains the instruction to install homebrew. After install homebrew, type the following commands in Terminal:
brew install gsl boost cmake qt5 doxygenqt5 or doxygen parameter can be removed from the command if you do not need to compile GUI tool or generate HTML documentation locally. It is possible to not install a separate CMake if you decide to use a CMake that comes with some of the IDEs.
Other methods to install and manage these packages include macports, which is fairly similar to the steps for homebrew, or install or compile them manually. Even they could be installing packages in different paths, CMake should be able to locate them automatically.
TBD
TBD
TBD
After all dependencies are installed, download and build the project as follows:
-
Download the source code repository by
git cloneor downloading and extracting the.ziparchive. -
In Terminal, navigate to the source code root path, the top folder containing
README.md. -
Execute the following commands to build all targets (excluding unit tests for Release build type) in
buildsubfolder:mkdir build cd build cmake ../ -DCMAKE_BUILD_TYPE=ReleaseReplace
-DCMAKE_BUILD_TYPE=Releasewith-DCMAKE_BUILD_TYPE=Debugto build with debug configuration. Note that debug configuration will have significant performance impact on certain steps of the workflow and should only be used in development. If the build type is not specified, the behavior is undefined.
CMake will automatically configure and compile the project, and the outcome is in build/ folder, either in Release/ or Debug/ subfolder depending on the build type. The executives (hsisomap_cli, demo, GUI tool (TBD)) and example files are in bin/ subfolder, and the static library libhsisomap.a is in lib subfolder.
There is currently no installation configuration. You will need to manually copy these outcomes (and header files from include/ folder of the source repository if you need to use the library in your code) into the path you want.
The official website contains full documentation of the source code. If you need a local copy, please use Doxygen to generate them locally.
-
In Terminal, navigate to the root path of the project.
-
Execute the following commands:
cd doc doxygen
The generated HTML and LaTeX documentation are in doc/html and doc/latex respectively. For HTML documentation, locate index.html for the home page.
The easiest way to build / debug / develop using the project is to use an IDE (Integrated Development Environment). The recommended IDE is CLion. It can open the project directly and compile without configuration. Just open the project by importing the project root folder.
The only caveat is that you will need to manually create the scheme to build the targets in Release mode, otherwise the performance might be impacted for certain steps of the workflow. For CLion, click on the target dropdown menu on the top-right of the window, select "Edit Configurations". Then select one target, and click "Copy Configuration" button on the top toolbar. Select the new target, and change the "Configuration" dropdown menu to "Release". Also change the name to be like this. Perform this action for each build target (excluding unit tests). Use "Release" configuration for production, and "Debug" for development.
In some versions of CLion, "Release" mode may not be presented in the dialog. This means you need to manually specify that you need this mode before editing the configuration. Go to "Preferences -> Build, Execution, Deployment -> CMake", click the "+" button below the "Generation" listbox. Then the "Release" item should be automatically added into the listbox.
Other IDEs that support CMake should also work but not tested.
This section specifies the usage of the commandline tool. For GUI tool, navigate to the GUI tool README. This is for people who want to use the program as a tool in their high-dimensional data analysis workflow.
The commandline tool only requires one arguments: the configuration file path. In Terminal:
./hsisomap_cli /path/to/configuration/file.jsonThe configuration file is standard JSON file that is a nested key-values and arrays. We don't use commandline arguments to set parameters directly since there are so many combinations and permutations of parameters and a configuration is the best way to set them. The key-values are human-readable so it is very easy to read, create, or edit. We do recommend use the GUI tool to facilitate this process and prevent misspelling. Note that the extension can be any name but we recommend the use of .txt or .json so that you can quickly open it using your default editors.
The full documentation of the configuration file is in the configuration file README (TBD). A minimal example of configuration file is as follows:
{
"type": "hsisomap_task_configuration",
"version": 0,
"tasks": [
{
"task index": 1,
"task description": "Tests",
"input": "/IMAGE/FILE",
"output root path": "/OUTPUT/ROOT/",
"landmark": {
"implementation": "subsets",
"count": 2000,
"noise model": "mnf",
},
"knngraph": {
"implementation": "fixed",
"k": 20,
"backend": "adjacency list"
},
"dijkstra": {
"implementation": "opencl"
},
"retained bands": -1,
"output file": "manifold"
}
]
}This section explains some of the examples in demo program. This is for people who want to use the library or code in their projects.
TBD
The full documentation of the project is generated by Doxygen. Click here to view the documentation. If you want to generate documentation locally, refer to "Build" section.
hsisomap is released under the GNU Public License V3.0.
Please cite hsisomap in your publications if it helps your research:
@inproceedings{jin2016mitigating,
title={Mitigating noise in global manifold coordinates for hyperspectral image classification},
author={Jin, Can and Bachmann, Charles M},
booktitle={SPIE Optical Engineering+ Applications},
pages={99760J--99760J},
year={2016},
organization={International Society for Optics and Photonics}
}