Tools for tracing leaf an stem motion from RGB video.
The typical pipeline for this process is:
- Generate dense UV flow maps using RAFT
- Generate traces over time starting from a subsampled 2D mesh of pixels
- Extract fastest moving traces to remove those generated from lighting and other noise
- Cluster traces using dynamic time warping
- Save and display results, which can be post-processed (e.g. sonification)
A high-level user interface for using this package is provided in the /launch directory.
See the in-code documentation provided in each launch file.
| Python Launch File | Description |
|---|---|
| gen_flow.py | generate UV flow maps with a pretrained RAFT model |
| gen_trace.py | generate trace CSVs and complementary images using UV flow maps |
| cluster_trace.py | cluster traces from gen_trace.py using dynamic time warping |
| draw_trace.py | annotate video frames with traces from gen_trace.py |
| imsort.py | access file name sorting and indexing tools |
clean_up.sh is also provided to conveniently erase all existing images and CSVs generated.
All files in dancing_plant/raft, dancing_plant/flow.py, and alt_cuda_corr/, along with the model downloaded via download_raft_model.sh, are directly adapted from the RAFT GitHub repository.
This project has been tested using Python 3.7 on Ubuntu 16.04.
- Change directories to this project's root and install the dancing-plant package.
pip install -e .
-
Install PyTorch with CUDA support. PyTorch version 1.6.0 and CudaToolKit 10.1 are tested to work with the following step.
-
Compile RAFT's custom CUDA extension which significantly reduces GPU memory requirements.
cd alt_cuda_corr && python setup.py install && cd ..
- Install other Python requirements.
pip install -r requirements.txt
(Arc: pip install -U future if the future package is missing)
- Download the RAFT model.
./download_raft_model.sh
cmd2 0.8.9 requires pyperclip, which is not installed. cmd2 0.8.9 requires wcwidth; sys_platform != "win32", which is not installed. torch 1.6.0 requires future, which is not installed.
Arc Instruction: infer1.arc.vt.edu Firs-time use: module load EasyBuild
module load CUDA/10.1.243-GCC-8.3.0 module load Anaconda3/2020.11 nvidia-smi ## make sure you see GPUs module save conda create -n pytorch python=3.7.0 unset PYTHONPATH source activate pytorch conda install pytorch==1.6.0 torchvision==0.7.0 cudatoolkit=10.1 -c pytorch export CXX=g++ git clone https://github.com/alanwangvt/dancing-plant.git "run steps 1,3,4,5 above" use "pip install" to install any missing packages
After first-time: module restore unset PYTHONPATH source activate pytorch
Alan Wang:
- Modify gen_flow.py to specify the work folder and the collections to be processed;
- Git push and pull to ~/dancingplant on arc;
- sbatch runGenFlow.sh; It will generate a raft-flow-raw folder in each image set;
- Wait untill step 3 is completed, in the dancingplant folder run: python launch/batchTrace.py, this processes all the traces and generates tracks, trace_cache, clusters, and annotations;
- run: python read_trace.py , this converts all tracks into a csv file that contains features (x-, y-, z- distances between adjacent time points and max/min/mean distances in each dimension) and a class lable (0-before treatment, 1-immeidately after treatment, 2-after treatment)