yiminglin-ai/MiVOLO

MiVOLO: Multi-input Transformer for Age and Gender Estimation

MiVOLO: Multi-input Transformer for Age and Gender Estimation, Maksim Kuprashevich, Irina Tolstykh, 2023 arXiv 2307.04616

[Paper] [Demo] [BibTex] [Data]

MiVOLO pretrained models

Gender & Age recognition performance.

Model	Type	Dataset (train and test)	Age MAE	Age CS@5	Gender Accuracy	download
volo_d1	face_only, age	IMDB-cleaned	4.29	67.71	-	checkpoint
volo_d1	face_only, age, gender	IMDB-cleaned	4.22	68.68	99.38	checkpoint
mivolo_d1	face_body, age, gender	IMDB-cleaned	4.24 [face+body] 6.87 [body]	68.32 [face+body] 46.32 [body]	99.46 [face+body] 96.48 [body]	model_imdb_cross_person_4.24_99.46.pth.tar
volo_d1	face_only, age	UTKFace	4.23	69.72	-	checkpoint
volo_d1	face_only, age, gender	UTKFace	4.23	69.78	97.69	checkpoint
mivolo_d1	face_body, age, gender	Lagenda	3.99 [face+body]	71.27 [face+body]	97.36 [face+body]	demo

MiVOLO regression benchmarks

Gender & Age recognition performance.

Use valid_age_gender.sh to reproduce results with our checkpoints.

Model	Type	Train Dataset	Test Dataset	Age MAE	Age CS@5	Gender Accuracy	download
mivolo_d1	face_body, age, gender	Lagenda	AgeDB	5.55 [face]	55.08 [face]	98.3 [face]	demo
mivolo_d1	face_body, age, gender	IMDB-cleaned	AgeDB	5.58 [face]	55.54 [face]	97.93 [face]	model_imdb_cross_person_4.24_99.46.pth.tar

MiVOLO classification benchmarks

Gender & Age recognition performance.

Model	Type	Train Dataset	Test Dataset	Age Accuracy	Gender Accuracy
mivolo_d1	face_body, age, gender	Lagenda	FairFace	61.07 [face+body]	95.73 [face+body]
mivolo_d1	face_body, age, gender	Lagenda	Adience	68.69 [face]	96.51[face]

Dataset

Please, cite our paper if you use any this data!

Lagenda dataset: images and annotation.
IMDB-clean: follow these instructions to get images and download our annotations.
UTK dataset: origin full images and our annotation: split from the article, random full split.
Adience dataset: follow these instructions to get images and download our annotations.
Click to expand!

After downloading them, your data directory should look something like this:
```
data
└── Adience
    ├── annotations  (folder with our annotations)
    ├── aligned      (will not be used)
    ├── faces
    ├── fold_0_data.txt
    ├── fold_1_data.txt
    ├── fold_2_data.txt
    ├── fold_3_data.txt
    └── fold_4_data.txt
```
We use coarse aligned images from faces/ dir.

Using our detector we found a face bbox for each image (see tools/prepare_adience.py).

This dataset has five folds. The performance metric is accuracy on five-fold cross validation.

images before removal fold 0 fold 1 fold 2 fold 3 fold 4

19,370 4,484 3,730 3,894 3,446 3,816

Not complete data

only age not found only gender not found SUM

40 1170 1,210 (6.2 %)

Removed data

failed to process image age and gender not found SUM

0 708 708 (3.6 %)

Genders

female male

9,372 8,120

Ages (8 classes) after mapping to not intersected ages intervals

0-2 4-6 8-12 15-20 25-32 38-43 48-53 60-100

2,509 2,140 2,293 1,791 5,589 2,490 909 901
FairFace dataset: follow these instructions to get images and download our annotations.
Click to expand!

After downloading them, your data directory should look something like this:
```
data
└── FairFace
   ├── annotations  (folder with our annotations)
   ├── fairface-img-margin025-trainval   (will not be used)
       ├── train
       ├── val
   ├── fairface-img-margin125-trainval
       ├── train
       ├── val
   ├── fairface_label_train.csv
   ├── fairface_label_val.csv
```
We use aligned images from fairface-img-margin125-trainval/ dir.

Using our detector we found a face bbox for each image and added a person bbox if it was possible (see tools/prepare_fairface.py).

This dataset has 2 splits: train and val. The performance metric is accuracy on validation.

images train images val

86,744 10,954

Genders for validation

female male

5,162 5,792

Ages for validation (9 classes):

0-2 3-9 10-19 20-29 30-39 40-49 50-59 60-69 70+

199 1,356 1,181 3,300 2,330 1,353 796 321 118
AgeDB dataset: follow these instructions to get images and download our annotations.

Click to expand!

Ages: 1 - 101

Genders: 9788 faces of M, 6700 faces of F

images 0 images 1 images 2 images 3 images 4 images 5 images 6 images 7 images 8 images 9

1701 1721 1615 1619 1626 1643 1634 1596 1676 1657

Data splits were taken from here

!! All splits(all dataset) were used for models evaluation.

images before removal	fold 0	fold 1	fold 2	fold 3	fold 4
19,370	4,484	3,730	3,894	3,446	3,816

only age not found	only gender not found	SUM
40	1170	1,210 (6.2 %)

failed to process image	age and gender not found	SUM
0	708	708 (3.6 %)

female	male
9,372	8,120

0-2	4-6	8-12	15-20	25-32	38-43	48-53	60-100
2,509	2,140	2,293	1,791	5,589	2,490	909	901

images train	images val
86,744	10,954

female	male
5,162	5,792

0-2	3-9	10-19	20-29	30-39	40-49	50-59	60-69	70+
199	1,356	1,181	3,300	2,330	1,353	796	321	118

images 0	images 1	images 2	images 3	images 4	images 5	images 6	images 7	images 8	images 9
1701	1721	1615	1619	1626	1643	1634	1596	1676	1657

Install

Install pytorch 1.13+ and other requirements.

pip install -r requirements.txt
pip install .

Demo

Download body + face detector model to models/yolov8x_person_face.pt
Download mivolo checkpoint to models/mivolo_imbd.pth.tar

wget https://variety.com/wp-content/uploads/2023/04/MCDNOHA_SP001.jpg -O jennifer_lawrence.jpg

python3 demo.py \
--input "jennifer_lawrence.jpg" \
--output "output" \
--detector-weights "models/yolov8x_person_face.pt " \
--checkpoint "models/mivolo_imbd.pth.tar" \
--device "cuda:0" \
--with-persons \
--draw

To run demo for a youtube video:

python3 demo.py \
--input "https://www.youtube.com/shorts/pVh32k0hGEI" \
--output "output" \
--detector-weights "models/yolov8x_person_face.pt" \
--checkpoint "models/mivolo_imbd.pth.tar" \
--device "cuda:0" \
--draw \
--with-persons

Validation

To reproduce validation metrics:

Download prepared annotations for imbd-clean / utk / adience / lagenda / fairface.
Download checkpoint
Run validation:

python3 eval_pretrained.py \
  --dataset_images /path/to/dataset/utk/images \
  --dataset_annotations /path/to/dataset/utk/annotation \
  --dataset_name utk \
  --split valid \
  --batch-size 512 \
  --checkpoint models/mivolo_imbd.pth.tar \
  --half \
  --with-persons \
  --device "cuda:0"

Supported dataset names: "utk", "imdb", "lagenda", "fairface", "adience".

Changelog

CHANGELOG.md

ONNX and TensorRT export

As of now (11.08.2023), while ONNX export is technically feasible, it is not advisable due to the poor performance of the resulting model with batch processing. TensorRT and OpenVINO export is impossible due to its lack of support for col2im.

If you remain absolutely committed to utilizing ONNX export, you can refer to these instructions.

The most highly recommended export method at present is using TorchScript. You can achieve this with a single line of code:

torch.jit.trace(model)

This approach provides you with a model that maintains its original speed and only requires a single file for usage, eliminating the need for additional code.

License

Please, see here

Citing

If you use our models, code or dataset, we kindly request you to cite the following paper and give repository a ⭐

@article{mivolo2023,
   Author = {Maksim Kuprashevich and Irina Tolstykh},
   Title = {MiVOLO: Multi-input Transformer for Age and Gender Estimation},
   Year = {2023},
   Eprint = {arXiv:2307.04616},
}

About

MiVOLO age & gender transformer neural network

Languages

Language:Python 98.9%Language:Shell 1.1%