Model fitting to videos
codr1 opened this issue · comments
Quick question on how things work - it seems that reset the landmarks every frame. So does the fitting improve with multiple frames? Does the PCA get refined? Or is it each frame for itself?
4dface is a very simple demo from around a year ago that performs per-frame fitting, as you correctly described. The fitting function can take initial parameters (for example from the previous frame), in which case it converges faster, but it's still per-frame (and I don't remember if I finished implementing this - but it should be easy to find out).
We also have a multi-frame fitting that fits to multiple frames at once, and thus improves the shape estimates. This is currently in a separate branch here, and we've got very promising results with it. It will eventually be merged to the eos master branch, but it'll take a while more.
@patrikhuber are you sure that multiframe fitting in this branch works because ig was updated a month ago?
@Thanh-Binh I'm not sure I understand your question. That is the multi-frame fitting that we have been using and it works very well for us.
@patrikhuber how many frames do you need for each fitting call?how about accuracy vs runtime?
@Thanh-Binh We determined 10-20 to be a good number. The accuracy increases quite steeply when going from 1..2..3..5..10 frames. From then on, it flattens a bit.
I can't give concrete runtime numbers from the top of my head but our implementation is pretty fast and IIRC it's something like linear in the number of images.
Very nice to hear! I will test it soon!
I'm going to close this as it's not an issue per se, and multi-frame fitting is being worked on in above linked branch in eos.