Questions? jmanley@rockefeller.edu
See the bottom for the original documentation provided by Cashman & Fitzgibbon to reproduce the results from their 2012 paper (http://ieeexplore.ieee.org/document/6165306/). The dolphin model produced by running the code described in their README.txt is given in fitzgibbonModel.mat.
This code was written and tested in MATLAB R2017a and requires the Optimization Toolbox.
Original source code retrieved from https://archive.codeplex.com/?p=forms.
Forms uses a procedure developed by Cashman & Fitzgibbon to fit a 3D morphable model to a collection of 2D images showing some type of object in its various poses and conformations. This fitting is formulated as an energy minimization problem, and the model represents the object as a linear combination of subdivision surfaces.
- set_initial_conditions.m GUI for easily inputting constraints and initial orientations for new datasets.
- Reformatting the forms.m output as a structure (instead of a vector with very careful indexing).
- Once a model has been built, fit_model_to_image.m allows fitting a model to another image outside the original input set.
- calculate_shapediff.m provides a potential metric for how well the model has fit the image's silhouette.
See run_forms_bananas_dolphins.m for fitting the model to the example data provided by Cashman & Fitzgibbon.
See run_forms_new_images.m for curating new images and fitting the model to a new dataset.
This algorithm requires a set of 2D images, in which the desired object has been segmented. While Cashman & Fitzgibbon stored their data in .ply and .fpj files (as seen in forms/projects), we have opted to simply store our data in structure arrays within .mat files. This is referred to as a Forms 'project'.
project.vertices -> a Nx3 matrix specifying the N vertices of the pre-defined mesh model
project.mesh -> a meshtri (see Cashman & Fitzgibbon's class definition)
project.faces -> a Nfx3 matrix specifying the indices of vertices which form the Nf triangular faces of the pre-defined mesh model
project.cand_ixs ->
project.cand_uvs ->
project.cand_limits ->
project.cand_dists ->
project.cand_derivs ->
For our analyses to date, the above fields have been the same as those utilized in Cashman & Fitzgibbon's dolphin analysis (see fitzgibbonModel.mat). The following fields should be specified for future analyses of new dolphins datasets:
project.images -> a Nx1 structure array containing the input dataset for N input images
project.parameters -> a structure array containing the parameters for running the model fitting in forms.m. See makeModelParameters.m.
project.images should contain the following fields for the i-th image project.images(i):
image -> the mxn(x3) original image
silhouette -> a Nx2 matrix containing the 2D location of N points along the object's silhouette
frame -> the frame index at which this image occurs in the movie
movie -> the path to this frame's original movie
constraints3d -> a Nx1 vector containing the indices of the N contraint points on the 3D mesh model. This index refers to an actual 3D location given in project.vertices.
constraints2d -> a Nx2 matrix containing the 2D locations of the N constraint points on the image, such that constraints3d(i) corresponds to the location of constraints2d(i,:) on the model.
constraintsonsil -> a Nx1 binary vector describing whether is constraint point is on (1) or off (0) the silhouette.
normalsLeft -> a Nx1 binary vector describing whether the normal vectors should be flipped (0) or not (1).
rotate -> a rotation matrix describing the approximate orientation of the dolphin for the model's initial conditions. This is specified by hand using set_initial_conditions.m.
translate -> a translation matrix for aligning the 3D model with the 2D silhouette, found using align_project.m.
scale -> a dilation matrix for aligning the 3D model with the 2D silhouette, found using align_project.m.
transform -> the full transformation matrix, such that transform = rotate * translate * scale.
points -> a 4x(N/4)x2 matrix containing a reshaped version of the silhouette, in order to be compatible with forms.m. % Why?
Once you have initialized a project with a mesh structure, images, and silhouettes, follow the pipeline below to find a 3D morphable model for your object.
Firstly, the initial conditions must be specified by running the following for each of the images i=1:length(project.images):
set_initial_conditions(project,i)
This will open a figure and guide you through setting the initial orientation of the object (using the angle sliders), selecting the constraint points, and properly orienting the silhouette normal vectors. Note: this also automatically runs the alignment code > align_project(project).
If you choose to use the default parameters, the following creates the parameter structure:
parameters = makeModelParameters; % see makeModelParameters or Cashman & Fitzgibbon's original paper for a description of these parameters
The model can then by found by running:
model = forms(project, parameters);
The fit of the model for image i can the be visualized by running:
plot_modelfit(project, i, model);
The norm of the ith basis shape can be plotted on the 0th mode by running:
plot_basisshapenorm(project, model, i);
See Cashman & Fitzgibbon's original README below. Note that the forms function has been modified in order to store the model in a structure, rather than in a carefully indexed single vector. The function archive/convert_modelVec_to_modelStruct can be utilized to convert the vector version of a model to the structure version of a model that is utilized here. In addition, the parameters inputs have been modified to be stored in a structure.
Forms: Flexible Object Reconstruction from Multiple Silhouettes
Thomas J. Cashman Andrew W. Fitzgibbon
University of Lugano Microsoft Research, Cambridge
This MATLAB code gives the implementation for our paper 'What Shape are
Dolphins? Building 3D Morphable Models from 2D Images'. You will need the
MATLAB Optimization Toolbox to run it.
To try the code on one of our datasets, load a Forms 'project', and then use the
function 'forms' to run our optimization. For example:
>> bananas = read_project('.\projects\bananas.fpj');
>> banana_model = forms(bananas, 2);
You can then compare the fit of the model before and after optimization, by
using
>> plot_modelfit(bananas, 1); % Show banana 1 before optimization
>> plot_modelfit(bananas, 1, banana_model); % Show banana 1 after optimization
Or take a look at the basis shapes plotted as a colour map by using
>> plot_basisshapenorm(bananas, banana_model, 1); % First basis shape norm
>> plot_basisshapenorm(bananas, banana_model, 2); % Second basis shape norm
To reproduce our results, use the weights and normal noise estimates described
in the paper, and use the function 'align_project' to find the camera
translation and scale parameters automatically, i.e.
>> bananas = read_project('.\projects\bananas.fpj');
>> bananas = align_project(bananas);
>> banana_model = forms(bananas, 2, 0.5, 0.25, 4);
>> pigeons = read_project('.\projects\pigeons.fpj');
>> pigeons = align_project(pigeons);
>> pigeon_model = forms(pigeons, 7, 0.25, 0.05, 5);
>> bears = read_project('.\projects\bears.fpj');
>> bears = align_project(bears);
>> bear_model = forms(bears, 10, 0.25, 0.25, 4);
Note that you should expect some of these optimizations to take a long time.
This is particularly true for the dolphins project, which can be used to build a
a model from 32 dolphin instances:
>> dolphins = read_project('.\projects\dolphins.fpj');
>> dolphins = align_project(dolphins);
>> dolphin_model = forms(dolphins, 8, 0.5, 0.25, 10);
The dolphin dataset is available here with manual segmentations, rather than the
grab cut silhouettes from Powerpoint 2010. This is the reason we suggest
sigma_norm = 10 above, rather than the value 40 / 3 that appears in the paper:
the manual segmentations have more reliable normals, so we can use this slightly
lower noise estimate. However, there is no significant difference between the
manual and automatic segmentations; the results are very similar.