Alternative training approaches to insightface models

[jadechip]

Hi and thanks for your amazing contribution.
I am curious if you have any thoughts or guidance on retraining the model without using the insightface models as their license is quite restrictive. For example would it be possible to use a different model to obtain the face landmark vector like media pipe? https://developers.google.com/mediapipe/solutions/vision/face_landmarker/index#models

Thank you for your time!

[Jiehui Huang]

Hi and thanks for your amazing contribution. I am curious if you have any thoughts or guidance on retraining the model without using the insightface models as their license is quite restrictive. For example would it be possible to use a different model to obtain the face landmark vector like media pipe? https://developers.google.com/mediapipe/solutions/vision/face_landmarker/index#models

Thank you for your time!

We have tried not using the features extracted by insightface, or using other structural features, but in terms of effect, insightface is currently the best.
Yes, the structural features extracted by insightface are relatively strict and can be replaced by landmarks. This is how InstantID.
Of course, it is not limited to the current facial feature extraction models. You can also try other more structural feature extraction methods to replace insight for model training and inference. You may get some better results. We look forward to discussing with you. .

[jadechip]

Thank you for your insightful reply. It seems like InsightFace is indeed still irreplaceable, I was looking into retraining the scrfd_10g_bnkps.onnx and glintr100.onnx from InsightFace checkpoints as the training code is available, but creating a big enough dataset without any license restrictions has been challenging 😰

[John D. Pope]

@jadechip - these guys built out a model for facial feature extraction leveraging mediapipe -

https://github.com/Zejun-Yang/AniPortrait/blob/cb86caa741d6ab1e119ea7ac2554eb28aabc631b/src/utils/mp_utils.py#L19
it seems like they did some extra work to extract to extract facial expressions
https://github.com/Zejun-Yang/AniPortrait/blob/cb86caa741d6ab1e119ea7ac2554eb28aabc631b/src/utils/face_landmark.py#L133