aistairc / FineBio

Data and code for the paper "FineBio: A Fine-Grained Video Dataset of Biological Experiments with Hierarchical Annotation"

FineBio: A Fine-Grained Video Dataset of Biological Experiments with Hierarchical Annotation

This repository includes the download instuction and the code for the paper FineBio: A Fine-Grained Video Dataset of Biological Experiments with Hierarchical Annotation.

Download

Usage is limited to non-commercial research/development use. Users need to sign the license form to access the videos, metadata and annotation. Please sign to the license agreement and submit via this form.

Link to the dataset and required credentials will be sent by e-mail after approval.

Dataset

Name convention

Video and annotation files are named by the following rules:

# First-person view
P<participant_id (1-32)>_<protocol_id (1-7)>_<take_id>.mp4
# Third-person view
P<participant_id (1-32)>_<protocol_id (1-7)>_<take_id>_T<camera_id (1-5)>.mp4
# Atomic operation annotation
P<participant_id (1-32)>_<protocol_id (1-7)>_<take_id>.txt
# Object detection images
P<participant_id (1-32)>_<protocol_id (1-7)>_<take_id>_<frame_num>.jpg
P<participant_id (1-32)>_<protocol_id (1-7)>_<take_id>__T<camera_id (1-5)>_frame_num>.jpg

Pre-trained Models

Please refer to the README for each benchmark for details.

Step segmentation

Backbone	Model	Acc	Edit	F1@10	F1@25	F1@50	F1@75	Weights
I3D	MS-TCN++	90.2	96.7	97.4	96.7	93.5	73.4	Link
I3D	ASFormer	87.2	94.8	94.2	92.7	86.5	67.0	Link

Atomic Operation Detection

Backbone	Model	mAP@0.3	0.4	0.5	0.6	0.7	Avg.	Weights
I3D	ActionFormer (multi-head + set classification)	45.2	41.7	36.5	28.4	18.7	34.1	Link

Object Detection

Model	AP	AP50	AP_manipulated	AP_affected	Weights
DINO	53.3	77.4	55.9	51.6	Link
Deformable DETR	56.1	78.5	64.0	58.8	Link

Manipulated/Affected Object Detection

	Hand	H + Manipulated	H + M + Affected	Weights
Left Hand	96.8	6.5	5.9	Link
Right Hand	94.5	22.2	10.7

Previous methods used in the implementation

Please cite the necessary methods if you used our baseline models.

• I3D
• RAFT
• MS-TCN++
• ASFormer
• ActionFormer
• Deformable DETR
• DINO
• Hand Object Detector

Support

If you find any problem, please report to Takuma Yagi (takuma.yagi[at]aist.go.jp) or by this repository's issue.

Citation

Please cite our work if you have used our data or code:

@article{yagi2024finebio,
  title={Finebio: a fine-grained video dataset of biological experiments with hierarchical annotation},
  author={Yagi, Takuma and Ohashi, Misaki and Huang, Yifei and Furuta, Ryosuke and Adachi, Shungo and Mitsuyama, Toutai and Sato, Yoichi},
  year={2024},
  journal={arXiv preprint arXiv:2402.00293},
}

License

The code is licensed under the MIT License.