Giters

hellodfan / GrainDet

The repository for paper titled with $GrainDet: An Efficient Dual-Camera System for Identifying Defective Kernels of Cereal Grains$

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Overview

This repository contains detailed information on our IEEE RA-L paper entitled AV4GAINsp: An Efficient Dual-Camera System for Identifying Defective Kernels of Cereal Grains.

Code

Train

All codes are stored in src folder, and the detailed train and test scripts can be found in the corresponding sub-folder README.md:

  • recognition (The recognition code of our proposed three-stage framework, thanks for TASN providing the trilinear attention code)
  • mmdetection (Faster RCNN, YoloX, Mask RCNN, RTMDet)
  • SAM

Evaluate

Please read src/eval/README.md.

Data

  • For single-kernel images, the dataset (processed oriented and single-kernel images) can be found (below).

  • For raw images, we have provided XML files in VOC format and JSON files in COCO format. It should be noted that the XML file corresponds to annotations for a single image, while the JSON file corresponds to annotations for two sides combined.You can also convert XML files to JSON files through parsexml.py and makecoco.py.

Pre-trained models

Data Acquisition

Overview

  • Sample Collection
    • Crops are harvested and transported into the grain centers
    • Raw grains are sampled to extract laboratory samples
    • Laboratory samples are further pre-processed by using handed tools, such as sieves, fans, tweezers
  • Image Acquisition
    • Grain kernels are fed into our device
    • Our framework can produce a number of single-kernel images
  • Image Annotation (during using our device, this stage is replaced by our Recognition model)
    • single-kernel images are annotated by six human experts

Acquisition Phase

Feeding Transporting Capturing Recycling
Paper
GIF

Image Data Analysis and Annotations

Our Analysis Framework

  • $I_{up}$ and $I_{do}$ are fed into our three-stage framework
    • 1, $I_{up}$ and $I_{do}$ are tiled into 15 overlapping patches, with each patch having 1500$\times$1500 pixels.
    • 2, Two raw images can produce a total of 30 patches
  • Stage-Detection:
    • 1, Each patch is fed into an object detection model ($\phi_{d}$, YoloX only doing detection without classification) to detect all kernel images
    • 2, All patches from $I_{up}$ can obtain $M_{up}$ kernel images that are saved into UPDetRes
    • 3, All patches from $I_{do}$ can obtain $M_{do}$ kernel images that are saved into DoDetRes
  • Stage-Pair:
    • All kernels in UPDetRes and DoDetRes are paired according to their centroid position and spatial distance, obtaining $M=\max(M_{up},M_{do})$ single-kernel images, with each single-kernel image has two views
  • Stage-Recognition:
    • 1, Two views in a single-kernel images are horizontally merged into an image
    • 2, Image is fed into the classification model ($\phi_{c}$, ResNet with imbalanced losses)

Image Data Annotations

  • We employ our framework (stage: detection and pair) to generate a vast number of candidate single-kernel images, and these images are used for building high-quality dataset: GrainDet.
  • we have engaged a team of six experienced annotators:
      1. Four senior inspectors with 5-10 years of inspection experience and currently working in authoritative quality inspection centers;
      1. Two experts with over 10 years of experience and holding leadership positions in the quality inspection center, who are also involved in the development of several national standards related to cereal grains.
  • The annotation for each kernel involves multiple steps to ensure consistency and accuracy:
      1. If all 4 senior inspectors assign the same label, then the label is confirmed (more than 99.9% of kernels are annotated in this step);
      1. In case where 3 of 4 senior inspectors assign the same label, then two experts are consulted to reach a final agreement and confirm the label;
      1. The remaining images are discarded.
  • To maintain the high quality of our dataset, we exclusively include samples from step1 and step2 samples in our GrainDet.

Dataset: GrainDet

Our dataset, called GrainDet, involves three types of cereal grains: wheat, rice and sorghum.

  • For wheat data, we annotated about $80K$ single-kernel images, including $48K$ normal grains and $32K$ images for six types of defective grains and impurities.
  • For rice data, we annotated about $24K$ single-kernel images, including $15K$ normal grains and $9K$ images for six types of defective grains and impurities.
  • For sorghum data, we annotated about $40K$ single-kernel images, including $24K$ normal grains and $16K$ images for six types of defective grains and impurities..

Category Abbreviation

Species Category Abbreviation
Wheat, Rice, Sorghum Normal grains NOR
Wheat, Rice, Sorghum Sprouted grains SD
Wheat, Rice, Sorghum Fusarium & Shrivele grains F&S
Wheat, Rice, Sorghum Broken grains BN
Wheat, Rice, Sorghum Grains attacked by pests AP
Wheat, Rice, Sorghum Impurities IM
Wheat Black point grains BP
Rice Unripe grains UN
Sorghum Heated grains HD

Examples

NOR SD FS BP/HD/UN MY BN AP IM
Wheat
Rice
Sorghum

Data distribution of defective kernels

NOR SD FS BP/HD/UN MY BN AP IM Total
Wheat 48.1K 5.1k 5.1k 1.4k 5.2k 5.8k 5.2k 4k 80K
Rice 16k 1.1k 1.2k 1.2k 1.2k 1.2k 1.2k 1.6k 24.7K
Sorghum 24k 3.6k 1.4k 0.2k 3.6k 3.6k 3.6k 1.2k 41K

Access the GrainDet

Wheat Images

Rice Images

Sorghum Images

AV4GAInsp vs. Human Experts Experiments

Calc_kappa.py Script

The script is used for obtaining the confusion matrix and kappa statistics.

calc_kappa.py

Access the Images for Kappa Tests

Detailed confusion matrixes between Devices and Human Experts

SampleID $S_1$ $S_1$ $S_2$ $S_2$ $S_3$ $S_3$
DeviceID $D_1$ $D_2$ $D_1$ $D_2$ $D_1$ $D_2$
$J_1$ Kappa_0.970 Kappa_0.902 Kappa_0.952 Kappa_0.907 Kappa_0.960 Kappa_0.949
$J_2$ Kappa_0.970 Kappa_0.930 Kappa_0.952 Kappa_0.915 Kappa_0.971 Kappa_0.949
SampleID $S_4$ $S_4$ $S_5$ $S_5$ $S_6$ $S_6$
DeviceID $D_1$ $D_2$ $D_1$ $D_2$ $D_1$ $D_2$
$J_1$ Kappa_0.949 Kappa_0.913 Kappa_0.925 Kappa_0.902 Kappa_0.895 Kappa_0.867
$J_2$ Kappa_0.953 Kappa_0.921 Kappa_0.937 Kappa_0.908 Kappa_0.904 Kappa_0.878

Citations

If our paper has been of assistance, we would appreciate it if you could consider citing it in your work.


@article{fan2023av4gainsp,
  title={AV4GAInsp: An Efficient Dual-Camera System for Identifying Defective Kernels of Cereal Grains},
  author={Fan, Lei and Ding, Yiwen and Fan, Dongdong and Wu, Yong and Chu, Hongxia and Pagnucco, Maurice and Song, Yang},
  journal={IEEE Robotics and Automation Letters},
  year={2023},
  publisher={IEEE}
}



@inproceedings{fan2022grainspace,
  title={GrainSpace: A Large-scale Dataset for Fine-grained and Domain-adaptive Recognition of Cereal Grains},
  author={Fan, Lei and Ding, Yiwen and Fan, Dongdong and Di, Donglin and Pagnucco, Maurice and Song, Yang},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={21116--21125},
  year={2022}
}

@article{fan2023annotated,
  title={An annotated grain kernel image database for visual quality inspection},
  author={Fan, Lei and Ding, Yiwen and Fan, Dongdong and Wu, Yong and Chu, Hongxia and Pagnucco, Maurice and Song, Yang},
  journal={Scientific Data},
  volume={10},
  number={1},
  pages={778},
  year={2023},
  publisher={Nature Publishing Group UK London}
}


@incollection{fan2023ai4graininsp,
  title={Identifying the Defective: Detecting Damaged Grains for Cereal Appearance Inspection},
  author={Fan, Lei and Ding, Yiwen and Fan, Dongdong and Wu, Yong and Pagnucco, Maurice and Song, Yang},
  booktitle={ECAI 2023},
  year={2023},
  publisher={IOS Press}
}

About

The repository for paper titled with $GrainDet: An Efficient Dual-Camera System for Identifying Defective Kernels of Cereal Grains$

Languages

Language:Python 76.4%Language:Jupyter Notebook 22.0%Language:Cuda 0.8%Language:C++ 0.5%Language:Shell 0.4%