jessecusack / Nortek_processing_tests

Tests of different data processing tools for Nortek ADCPs

Nortek Processing Tests

Tests of different data processing tools for Nortek ADCPs

Requirements

• Conda package manager (I recommend the lightweight version miniconda)
• MATLAB, including one of the toolboxes below:

 'quaternion' requires one of the following:
  Automated Driving Toolbox
  Navigation Toolbox
  Radar Toolbox
  Robotics System Toolbox
  Sensor Fusion and Tracking Toolbox
  UAV Toolbox

Installation

Run the shell scrip:

./install_environment.sh

More info at the bottom. Look at the script itself for the manual step-by-step.

Test data

Eli shared a Box drive containing some test data. I think this is from an instrument on a glider. To run the code you need to copy the following files to data/external:

Box/LeConte/testdata/Data_0000.ad2cp  # 1 GB 
Box/LeConte/testdata/Data_0000.ad2cp.00001_1.nc  # 330 MB 
Box/LeConte/testdata/Data_0000.ad2cp.00001_2.nc  # 330 MB 

My understanding is that the netcdf files are generated from the ad2cp file.

Processing Tools

ocean-tools (MATLAB)

Dylan's toolbox is available at https://github.com/dswinters/ocean-tools. It is downloaded automatically using the script matlab_toolboxes/get_toolboxes.sh.

I tested the toolbox by first parsing the raw Nortek data and converting to a .mat with:

scripts/ocean_tools_convert.m

Then I tried rotating into earth coordinates:

scripts/ocean_tools_rotate.m

Then I plotted some of the data:

scripts/ocean_tools_check_data.m

This requires a toolbox that provies the quaternion function.

Notes

• The parsed data does not contain pressure or temperature, but I find them in the netcdf? Not sure if these are in the ad2cp file or added somewhere else in the netcdf creation.
• Requires additional MATLAB toolboxes for quaternion.
• Rotation worked, I think...
• Looks like a good MATLAB solution

xarray (python)

Installed with the conda environment. I will use it to read the Nortek generated netCDF files. This is done in the notebook scripts/xarray_check_data.py. To convert this script to a notebook, you may need to use:

conda activate nortek-test
jupytext --to=ipynb scripts/xarray_check_data.py

This is done automatically by the install script.

Notes

• Data is stored in the group Data/Burst.
• Obscene variable and coordinate layout, eurgh.
• Data appear to be in beam coordinates.

dolfyn (python)

Installed with the conda environment.

scripts/dolfyn_check_data.ipynb

Notes

• Loads the ad2cp data no problem.
• Can do beam rotations, but was confused by the Glider model of ADCP.
• Reads lots of additional data and metadata inc. temperature and pressure
• Looks very promising for a python solution.

Installing and removing the environment

A conda environment is specified in environment.yml and may be install using the appropriate bash scripts.

To install:

./install_environment.sh

To remove:

./remove_environment.sh

These also install/remove the jupyter kernel for the environment.

If these don't execute, you might need to change the file permissions with chmod u+x *.sh.

Project Structure

Nortek_processing_tests/
    ├── LICENSE
    ├── README.md          <- The top-level README for people using this project.
    ├── data/
    │   ├── external       <- Data pulled in from outside of this project.
    │   └── internal       <- Data generated within this project.
    │
    ├── scripts/           <- MATLAB scripts and jupyter notebooks
    │   └── README.md      <- Any information about the code, such as execution order. 
    │
    ├── figures/           <- Saved figures generated during analysis.
    │
    ├── environment.yml    <- Conda environment specification. Install using the bash scripts.
    │
    ├── matlab_toolboxes/  <- A place for 3rd party MATLAB toolboxes.
    │   ├── toolbox/
    │   └── get_toolbox.sh <- Script to download toolboxes.

---

• Free software: MIT license