ButterflyNet for Automating Laminar-Turbulent Flow Segmentation

The paper: https://link.springer.com/article/10.1007/s00348-022-03516-4
The project website: https://ridvansalihkuzu.github.io/butterflynet/
The project codebase: https://github.com/ridvansalihkuzu/butterflynet

MAIN OBJECTIVE OF THE PROJECT
PART 1: INSTALLATION AND RUNNING
PART 2: FURTHER DETAILS ABOUT THE CONDUCTED EXPERIMENTS
CONCLUSION
CITATION

1. Main Objective of this Project

Specifically, detection of laminar and turbulent flow regions and transition locations in between are of crucial interest in a range of aviational applications, since the achievement of a fuel-efficient wing, airfoil or rotor blade design and reducing the instability caused by drag forces are strongly related with the way of handling laminar and turbulent flow exposed by the wing and blade surfaces.

As seen in Figure 1, Step 4 in the aircraft body component design workflow is the expert investigation to determine the laminar-turbulent regions on the thermoghraphic measurement images. However, those investigations might be entirely manual, or semi-automatic. That is why, The main objective of this project is to provide a reliable automation method for facilitating the localisation of laminar-turbulent flow regions by using recent artificial intelligence approaches.

Figure 1: High-level workflow for building new wings, stabilizers or blades (Please note that the workflow ignores some of the side steps which are out of context for this work).

2. Installing and Running the User Interface

2.a. Installing the User Interface for Automatic Laminar-Turbulent Flow Localization

As you can see below, there is a user interface called Annotation Window which can be used to handle checking the quality of the predictions for flow-localization and to make some postprocessing for removing artifacts or holes in the segmented thermographic measurement:

Figure 2: Annotation Window for Automatic Flow Localization on Thermographic Measurements.

In order to prepare this user interface, there are two different options:

OPTION: The following commands can be used directly to run the python file:
```
 $ cd butterflynet
 $ python -m ui.annotation_window
```
OPTION: An executable file can be generated and utilized independently with the following command:
```
$ cd butterflynet/ui
$ pyinstaller  --onefile  annotation_window.py --hidden-import 'skimage.filters.rank.core_cy_3d'
```
This command creates an executable file dist/annotation_window.exe which can launch the user interface.

2.b. Running the User Interface for Automatic Laminar-Turbulent Flow Localization

Annotation window can be launched with the following command directly inside Python:

 $ cd butterflynet
 $ python -m ui.annotation_window

or it can be run via the generated executable:

 $ cd butterflynet/ui/dist/
 $ annotation_window.exe

In this interface:

Input Directory is the directory where the images to be segmented are located
Output Directory is the output directory where the predicted mask will be saved
Model Directory is '.tf' formatted tensorflow model obtained after standard supervised learning or supervised fine-tuning.

When those directories are defined, Start Processing button should be clicked to run the process. After that, evaluation of the prediction quality or post-processing of the predicted masks can be possible as illustrated in the video above.

3. Editing the User Interface

The UI can be edited or updated via QT Designer if requred in certain circumstances, such as adding new functionalities to the UI. To do this:

Install QT Designer:
```
$ pip install PyQt5Designer
```
Run QT Designer:
```
$ sudo designer
```
On the opened editing tool, you can load ui/gui.ui file to see the drag-drop functionalities of the existing interface.
There, you can edit ui/gui.ui file and save it again with updated buttons, labels, components, etc.
And finally the new Annotation Window can be compiled with the following command:
```
$ cd butterflynet/ui
$ sudo pyuic5 "gui.ui" -o "gui.py"
```
After those steps, do not forget to check compatibility of ui/gui.py with ui/annotation_window.py.

4. Folder Structure for Training and Evaluation

In this work, Adaptive Attention Butterfly Network (shortly ButterflyNet) has been proposed for the effective separation of laminar flow from the other flow regions. The proposed ButterflyNet falls into the category of Ensemble U-Nets with the inspiration from the following studies:

Attention U-Net which increases the model sensitivity and prediction accuracy with minimal computational overhead,
DoubleU-Net which is useful to eliminate the small artefacts in the segmented areas,
and nnU-Net which enables dynamic adaptation of network topologies to different image modalities and geometries, and compensates the issue of having a large number of network parameters and of inefficient throughput time in case of using cascaded networks as similar to our proposed ButterflyNet.

Figure 3: The Adaptive Attention Butterfly Network (ButterflyNet) architecture (top), and the details of the blocks utilised in the ButterflyNet (bottom) .

In this project, the following benchmark U-Net models have been compared with the proposed ButterflyNet:

Regarding the Experimental Part of the Project:

In folder experimental/, you can find:
- main.py as the main training script to compare different U-Net architectures,
- model_selector.py for orchestrating the U-Net model selection and initialisation,
- custom_data_generator.py for data reading for training and inference,
In folder model_zoo/adaptive_net_family, you can find the proposed ButterflyNet and its utilities,
In folder model_zoo, you can find also some other U-Net architectures applied for benchmark comparison in this work,
The U-Net architectures not found in these folders were inherited directly from the python package keras-unet-collection,

Regarding the Stable Release of the Project:

In folder general/, you can find:
- main.py as the main training script to compare different U-Net architectures,
- model_zoo.py for orchestrating the U-Net model selection and initialisation,
- custom_data_generator.py for data reading for training and inference,
- custom_losses.py for defining some segmentation loss functions, customized for this work, or not included in tensorflow/keras framework,
- custom_metrics.py for defining some segmentation performance metrics, customized for this work, or not included in tensorflow/keras framework,
- post_processing.py for some image processing functions utilized in the user interface,
- utils.py for defining general utility fucntions applied during training and inference on U-Nets,

5. Preparing the Training Environment

For preparing the installation environment, there are two different options:

OPTION: conda environment can be built by installing the dependencies listed in docker/requirements.txt
```
$ cd butterflynet/docker
$ pip install -r requirements.txt
```
OPTION: docker environment can be built by running the following commands:
```
$ cd butterflynet/docker
$ docker build -t tensor_image .
```

For training and evaluation there are multiple options:

A supervised model for the binary segmentation can be directly trained if there exist sufficient number of input and mask images for training.
If the labelled data is not sufficient for model generalization, causing overfitting or high bias & variance between training and validation:
1. A self-supervised model can be trained on the unlabelled data, as a first step,
2. Later, this self-supervised model can be fine-tuned on the labelled data.

Those procedures will be detailed below:

6. Supervised Learning for Flow Segmentation

For training the evaluation, there are three different options:

OPTION: general/main.py script can be called for training as illustrated in the following command:
```
$ cd butterflynet
$ python -m general.main  --learning-rate 0.000025 --batch-size 64 --num-augment 8 --out-dir modeldir/supervised/ --pair-csv dataset/supervised/fileNames.txt --model-type 6 --data-dir dataset/supervised/image/ --label-dir dataset/supervised/mask/
```
As seen in the command, there are many arguments to be passed into the main.py script, the definitions of these arguments are as follows:
- BATCH_SIZE: number of batches; e.g. 32 and 64 are recommended as batch size
- CUDA: index of GPU sources; e.g. _--cuda '0' _ for single GPU or --cuda '1,2,3' for three different GPUs
- NUM_WORKERS: number of workers for loading the training data; default:2 and no need to change in most cases
- NUM_EPOCHS: number of epochs in training; default:180 and no need to change in most cases
- START_EPOCH: epoch index for resuming the training; default:0 which means the training starts from scratch
- LEARNING_RATE: learning rate of the optimizer selected for the training
- NUM_CLASSES: number of classes in segmentation; default:2 and other options are not supported for now
- NUM_AUGMENT: number of augmentation; e.g. --num-augment 8 is sufficient for standard supervised learning if the data size is less than 500
- MODEL_TYPE: model architecture index; 0: MobileUNet, 1: UNet, 2: ResUNet, 3:AttentionUNet with ResNet, 4:AttentionUNet with EfficientNet-B0, 5:AttentionUNet with EfficientNet-B7, 6:Attention UNet without backbone
- DATA_DIR: directory for input images; e.g. --data-dir 'dataset/supervised/image/'
- LABEL_DIR: directory for input labels; e.g. --data-dir 'dataset/supervised/mask/'
- PAIR_CSV: it is a pair list of input and label images; e.g. --pair-csv 'dataset/supervised/fileNames.txt' , it is mandatory for supervised learning
- WEIGHT_FILE: a weights file for the model initialization with pretrained weights; it should be --weights 'None' for standard supervised training
- OUT_DIR: directory for the trained output models with '.tf' extension
- LOG_FILE: a path to the log file to store the training, validation and test results for each run; default:'performance-logs.csv'
- EVALUATE: if it is True, only model evaluataion is run
- SELF_SUPERVISED: it determines if training mode is supervised or self-supervised; FALSE for standard supervised learning
- SELF_RECONSTRUCT: it determines if self-supervised learning is based on SimCLR or same-input reconstruction; FALSE for standard supervised learning
- TEMPERATURE: it is a self-supervised learning parameter for cost calculation, not used in the standard supervised learning
OPTION: general/main.py script can be called in a bash file as can be seen in script_train_supervised_general.bat:
```
$ cd butterflynet
$ bash script_train_supervised_general.bat
```
Here, please check if the parameters in the bash file are relevant for your run.

OPTION: If you configured the Docker environment, you can also call the above mentioned scripts inside the Docker as similar to the following example:

$ sudo docker run -it --init --rm --shm-size=32772m --gpus '"device=1,2,3,4"' -v "/<YOUR-HOME-DIR>/butterflynet/:/app" tensor_image bash script_train_supervised_general.bat

$ sudo docker run -it --init --rm --shm-size=32772m --gpus '"device=1,2,3,4"' -v "/<YOUR-HOME-DIR>/butterflynet/:/app" tensor_image python -m general.main  --learning-rate 0.000025 --batch-size 64 --num-augment 8 --out-dir modeldir/supervised/ --pair-csv dataset/supervised/fileNames.txt --model-type 6 --data-dir dataset/supervised/image/ --label-dir dataset/supervised/mask/

7. Self-supervised Learning and Supervised Fine-tuning

Figure 4: The architecture for self-supervised learning. Note that the augmented views are originated from either same or different inputs by random switching .

The aim of the self-supervised learning is to mitigate the lack of labelled data. In case of sufficient number of labelled data, this step is not necessary. However, when there is an overfitting issue or similars due to the lack of data, this approach benefits from:

either self-supervised learning based on SimCLR approach as illustrated in Figure 4,
or self-supervised learning based on reconstruction of same image at the output.

After the self-supervised learning on the unlabelled data, fine-tuning is made with the labelled data. The details of these steps are as follows:

7.a. Self-supervised Learning

Some reminders for the parameters of self-supervised learning:

DATA_DIR and LABEL_DIR should be the same directory, because the data does not have labels
PAIR_CSV should be 'None'
SELF_SUPERVISED should be TRUE
SELF_RECONSTRUCT should be TRUE for self-supervised learning based on reconstruction, FALSE for self-supervised learning based on SimCLR
TEMPERATURE can be set to any positive float; default:1.0
NUM_AUGMENT: can be set to 1, since the total number of unlabelled data is already huge

Considering those parameters, a sample call for the self-supervised learning is as follows:

$ python -m general.main  --model-type 6 --temperature 0.01 --learning-rate 0.000100 --batch-size 64 --num-epochs 120 --num-augment 1 --self-supervised --out-dir modeldir/unsupervised/ --data-dir dataset/unsupervised/image/ --label-dir dataset/unsupervised/image/

or, in case of Docker usage:

$ sudo docker run -it --init --rm --shm-size=32772m --gpus '"device=1,2,3,4"' -v "/<YOUR-HOME-DIR>/butterflynet/:/app" tensor_image python -m general.main  --model-type 6 --temperature 0.01 --learning-rate 0.000100 --batch-size 64 --num-epochs 120 --num-augment 1 --self-supervised --out-dir modeldir/unsupervised/ --data-dir dataset/unsupervised/image/ --label-dir dataset/unsupervised/image/

7.b. Supervised Fine-tuning

The fine-tuning followed by self-supervised learning should be done as follows:

Set the parameters for standard supervised learning as explained in Section 2.a Supervised Learning. However, consider the following parameters:

WEIGHT_FILE: should be the model directory created by self-supervised learning, e.g. '--weight-file modeldir/unsupervised/model_best_lr_5e-05_ty_6_ba_64.tf'
SELF_SUPERVISED: it should be FALSE in fine-tuning
SELF_RECONSTRUCT: FALSE if the WEIGHTS are based on SimCLR approach, TRUE if the weights are based on reconstruction of same image at the output

Thus, a sample call for fine-tuning on the labelled data:

   $ python -m general.main  --learning-rate 0.000025 --batch-size 64 --num-augment 8 --out-dir modeldir/supervised/ --pair-csv dataset/supervised/fileNames.txt --model-type 6 --data-dir dataset/supervised/image/ --label-dir dataset/supervised/mask/ --weight-file modeldir/unsupervised/model_best_lr_5e-05_ty_6_ba_64.tf

or, in case of Docker usage:

$ sudo docker run -it --init --rm --shm-size=32772m --gpus '"device=1,2,3,4"' -v "/<YOUR-HOME-DIR>/butterflynet/:/app" tensor_image python -m general.main  --learning-rate 0.000025 --batch-size 64 --num-augment 8 --out-dir modeldir/supervised/ --pair-csv dataset/supervised/fileNames.txt --model-type 6 --data-dir dataset/supervised/image/ --label-dir dataset/supervised/mask/ --weight-file modeldir/unsupervised/model_best_lr_5e-05_ty_6_ba_64.tf

7.c. Standalone Evaluation on Terminal

Sometimes there may be a need for evaluating the output of single input image, or outputs of images in a directory. In this case, the following command-line call can be useful:

 $ python -m general.evaluate  --input-dir dataset/supervised/image  --label-dir dataset/supervised/mask/ --out-dir dataset/supervised/predicted_supervised --weight-file modeldir/unsupervised/model_best_lr_5e-05_ty_6_ba_64.tf

Here, the parameters have the following roles:

INPUT_DIR: path to the input image folder or image file
LABEL_DIR: path to the input label folder or label file
OUT_DIR: path to the output image folder or image file
WEIGHT_FILE: a weights file for the model initialization with pretrained weights; it should NOT be --weights 'None'
SELF_SUPERVISED: it determines if evaluation mode is for supervised or self-supervised learning; FALSE for supervised learning

8. About Laminar-Turbulent Flow Segmentation

Laminar-Turbulent flow is a fluid dynamics phenomenon that refers to the motion of particles as they move through a substance. The distinction between laminar and turbulent flow is eminently important in aerodynamics and hydrodynamics because the type of flow has a profound impact on how momentum and heat are transferred. In principle, when particles of fluid have a property of a streamlined flow, and they follow a smooth path without interfering with one another, it is called laminar flow. On the other hand, turbulent flow means the chaotic and rough movement of particles through a substance , such as mixing and shifting between the flow layers and whirlpool-like patterns as illustrated in Figure 5.

Figure 5: Airflow over a wing exhibiting transition from laminar to turbulent.

8.a. Measurement Setup for Observing Flows on the Body of Aircraft

Measurement setups may differ from use case to use case. However, at least three things are shared between all setups: i) a surface, ii) an airflow to which the surface is exposed to, and iii) at minimum, one IR camera observing the surface.

A simplified laminar-turbulent flow measurement setup applied in the variety of our experiments can be described as follows:
A surface of an object is observed by an IR camera. The camera together with its optical system (lenses) spans a field of view (FoV) onto the object surface. Everything on this FoV is visible within the recorded image. As camera and object have their own reference system, coordinate systems for the object and camera exist. Modern IR cameras work with a focal plane array (FPA) being the sensor, hence the image plane is put into the sensor plane. Perpendicularly from the camera sensor plane, a view axis can be defined, which is the optical axis. The intersection of the optical axis with the surface is the focal point of the camera. Reference markers located in the FoV allow the image transformation from image to object coordinates. Since the work presented here is based on images in the aerodynamic field, the object of interest is exposed to a flow with a dedicated flow direction. Eventually, a laminar and turbulent region may develop inside the FoV if the object (flight-wings and vertical stabilizer in our experiments) is exposed to a flow.

Figure 6: A simplified laminar-turbulent flow measurement setup.

To Elaborate on one of the particular test cases, for AfLoNext project, two IR cameras had been installed into the horizontal tail plane (HTP) of the test aircraft, which was an AIRBUS A320. One IR camera was installed at the port (left hand) and one at the starboard (right hand) side of the aircraft. Both cameras were orientated in such a way that they observed a ROI at the vertical tail plane (VTP). As a consequence, IR images could be taken during flight simultaneously both from the left- and right-hand side of the VTP, as illustrated in Figure 7.

Figure 7: AFLoNext, digital mockup field of view (FoV) study using 20° lens.

8.b. Collected Thermoghraphic Measurement Examples

As shown in Figure 8, some thermoghraphic measurement samples contain scars, shadows, salt & pepper noises and contrast burst regions, demonstrating that realistic laminar-turbulent flow observation scenarios are subject to high noise. Besides, a laminar flow area may occur brighter or darker as compared to the regions in a turbulent flow. Due to some effect (e.g. shadowing the sun) it is even possible that, in one part of the image, the laminar flow area appears darker, and in another part, it appears brighter than the turbulent flow area.

Figure 8: Thermographic measurement examples from wind tunnel and flight test experiments: i. top and bottom row: wind tunnel ii. center row: vertical stabilizer from AFLoNext Project. Note that the red flow-separation lines are ground-truth boundaries upon the measurements.

9. Comparing ButterflyNet with Benchmark U-NETs

In order to demonstrate the effectiveness of our proposed flow segmentation architecture, the quantitative and qualitative comparisons among the benchmark architectures have been conducted after running the tests 20 times for each architecture. As summarised in Figure 9, ButterflyNet has excelled all the segmentation architectures in all evaluation metrics.

Figure 9: Kernel density estimations drawn by the segmentation scores taken from each test run in terms of the evaluation metrics PA, IoU and SDC.

Besides, as shown in Figure 10, the ButterflyNet is computationally less expensive than 4 of the benchmarks with total number of 15:11 Giga-Flops, and more memory efficient than 3 of the benchmarks with 24:64 million parameters.

Figure 10: Sørensen-Dice coefficient vs. computational complexity where G-FLOPs stands for the number floating-point operations required for a single forward pass, and the size of each ball corresponds to the model complexity.

Figure 11: Qualitative comparison of benchmark architectures. In the ground truth, the laminar flow regions are denoted as white (positive condition), while the rest is black (negative condition). Similarly, in the comparison of architectures, TPs are white, TNs are black, FPs are magenta and FNs are cyan.

10. Examining Cascaded Structure of ButterflyNet

Since the ButterflyNet has comprised two cascaded networks, WING1 and WING2 as seen in Figure 3, quantitative comparison among the outputs of them has been done to examine if the cascading had a positive impact in flow segmentation.

The qualitative analysis in Figure 12 verifies that masking the input with the output of WING1 and later feeding it into WING2 could eliminate some unwanted artefacts when automatically localising the separation boundary of laminar flow from turbulent flow.

Figure 12: Qualitative comparison of WING1 and WING2 in terms of flow segmentation. Note that WING2 has generated less artefacts than WING1 where the false positives are magenta and false negatives are cyan.

11. Conclusion

In this project, we handled the automatic separation of different flow regions over flight-wings and stabilizers using the thermographic flow observations and applying deep learning techniques, because detection of flow distribution has crucial importance for optimising the wing & stabilizer geometry and improving the flight efficiency. Since the laminar-turbulent flow measurements are usually exposed to high noise and variance across different thermographic observations, the existing efforts for the automation of flow localisation in the literature had lack of reproducibility, or they only achieved semi-automation with a strong dependency on human experts for guidance and correction.
To overcome such difficulties, we introduced a novel encoder-decoder architecture, namely ButterflyNet, for the automatic segmentation of flow regions. In order to compensate for the lack of manually labelled data caused by the time-consuming nature of analysis on IR thermography samples, we customised a self-supervised strategy and, in this way, we benefited from diverse sets of raw thermographic observations to improve the robustness of flow segmentation. The proposed approach achieved 97.34% pixel accuracy and 91.17% intersection-over-union in the automatic separation of laminar flow regions from the remaining regions.

12. Citation

In order to cite this study:

@article{butterflynet2022,
  title={Automatic Separation of Laminar-Turbulent Flows on Aircraft Wings and Stabilizers via Adaptive Attention Butterfly Network},
  author={Kuzu, R{\i}dvan Salih and Mühlmann, Philipp and Zhu, Xiao Xiang},
  journal={Experiments in Fluids},
  volume={63},
  number={10},
  pages={1--22},
  year={2022},
  publisher={Springer}
}

ridvansalihkuzu / butterflynet