Automatic Labeling of Motion Capture Markers

These scripts take care of automatically labeling the data collected from markers of a motion capture system, according to the names given in the first frame.

This code was used in the How We Type project. If you use it for your research then please remember to cite:

Anna Maria Feit, Daryl Weir, and Antti Oulasvirta. 2016.
How We Type: Movement Strategies and Performance in Everyday Typing. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 4262-4273. DOI: http://dx.doi.org/10.1145/2858036.2858233

Note: the following instructions can also be found in the Jupyter Notebook README.ipynb

Data Format

For the scripts to work, the motion capture files must fulfill the following criteria (see example log file):

comma separated value (csv) files
the first 6 lines are header lines where the 4th line contains the marker names, replicated 3 times each, starting from the third column on
the following lines contain the x-y-z data of each marker, preceded by the frame number and time

Working principle

The labeling is based on a nearest neighbor approach with some additional heuristics and features to make it more robust. In every frame a point is labeled according to the nearest marker from the previous frame independent of its given label, but only if:

there is no other point closer to that marker, otherwise it's marked as missing for that frame
the point is within a threshold distance of the marker, which is higher if the marker was missing in the previous frame
the point is within a threshold distance of the bounding box around all markers in the previous frame (ignores noise reflections in the tracking environment)

Missing data and skeleton

A Skeleton can be defined by assigning maximum 1 child and 1 parent marker to each marker. The skeleton is used to extrapolate the position of a missing marker based on the previous frame. The extrapolation computes two new positions based on the new position of the child and parent markers and the previous vectors between the child/parent and the marker in the previous frame, and averages between those 2 positions.

If a marker is missing, it is assumed to stay at the previous position, if no child or parent markers are available.

Hardcoded heuristics for hands

Finally, there are 2 hardcoded heuristics for correcting swapped markers IF the collected data comes from hands and the marker names correspond to names such as "Hands_R_L4", where:

each finger has 4 markers on each of the 3 joints and the fingertip
all names start with "Hands_"
the first single letter is R or L, corresponding to the Right or Left hand
the second letter is one of T, I, M, R, or L, correspnding to the Thumb, Index, Middle, Ring and Little finger
the number is between 1-4 and increases from the MCP joint (1) to the fingertip (4)

The heuristics detect cases where markers of neighboring fingers are swapped, or the fingertip and DIP markers on one finger. In the first case, the corresponding bones (lines between a parent and child marker) of the two fingers cross each other in 3D space (or are very close together), in the second case the fingertip is bend backwards in an unnatural way and the joint angle of the DIP joint is very small (can't be smaller than 90 degree).

Validation and manual labeling

The labeled data is plotted every 10000 frames and the plot is saved for inspection.

If the automatic process fails, it can be supported manually in 3 ways:

Marker mappings can be defined at a frame level. If the name for a certain marker at a certain frame is known (e.g. from inspecting the recording) it can be specified as an argument to the labeling process. Only for this frame the algorithm will rely on the name given by the user and label the corresponding marker data accordingly.
The user can manually label all markers in a single frame and define those frames as fallback frames. For those, the user guarantees that all present markers are given the correct label. Then the automatic labeling relies on the given label and continues from that point as described above.
Fully labeled markers can be defined, for which the user guarantees that they are correctly labeled throughout the whole take. Every point labeled with the name of the marker in the motion capture file, is labeled as such by the algorithm and not based on the nearest neighbor approach.

Running the scripts

To run the labeling process for a given file, simply create a new Take object with the filename as the only required argument. It takes care of:

Labeling the given data
Writing the labeled data to a new csv file in the same format
Plotting the data every 10000 frames and saving the file in the IMG folger for inspection

In addition, the following arguments can be defined to modify the labeling process as described above (default values given):

marker_names = []
A list of the marker names that should be used for labeling and that were used to label the data in the very first frame. If this is set to [] the script will try to figure them out automatically by taking only those names that do not start with "Marker" (e.g. Marker_27916).
frame_marker_names = {}
A dictionary of frames to a mapping from marker name to marker name. Specifies for a marker which label it corresponds to at the given frame.
fallback_frames = []
A list of fallback frames at which the captured data is labeled correctly
labeled_marker_names = []
A list of marker names that the algorithm can assume to be labeled correctly throughout the complete file
check_hand_skeleton_heuristics = 0
Binary value indicatinf if the heuristics for checking for swapped hand markers should be used. Set only to 1 if the marker are named as described above
use_skeleton = 1
Binary value indicating if a skeleton should be used. If set to 1, the parent-child relations must be adapted in the file skeleton.py according to the given marker names. See the file for further info.
debug = 1
Binary value indiciating if the saved plots should indicate the names of the markers relabeled at that frame
ignore_marker_names = []
A list of marker names that should be ignored during the relabeling (e.g. markers put for reference or on devices)
plot_every_X_frames = 10000
Integer that defines the frame interval for plotting the data for manual inspection. Data is plotted for the first and last frame and then every plot_every_X_frames frames.
plot_xlim, plot_ylim, plot_zlim = (-0.5,0.5)
Tuple with lower and upper limit of x,y, and z axis for plotting the motion capture data

The labeled logfile is saved to the same folder as the given logfile and marked as "_labeled". The reference images are saved to the IMG folder

from Take import *


# the only required argument is the filename:
logfile = "Logfiles/test.csv"

# the following arguments can be given to control the labeling process, see the file Take.py for further description. 
# Here: examples how to specify 
marker_names = []
frame_marker_names = {100:{"Hands_L_L4":"Hands_L_R4", "Hands_L_L3": "Hands_L_L2"}} 
fallback_frames = [2,3] 
labeled_marker_names=["Hands_L_L1"]
check_hand_skeleton_heuristics = 0
use_skeleton = 1
debug = 1
ignore_marker_names = ["Hands_K_right_top", "Hands_K_right_bottom", "Hands_K_left_top"]
plot_every_X_frames = 10000

# Create the Take object, which takes care of labeling, plotting and writing the new logfile. 
# Note that only the first argument is required and all others are optional. For the default values, simply write
# t = Take(logfile)

t = Take(logfile, 
         marker_names = marker_names,
         frame_marker_names = frame_marker_names,
         fallback_frames = fallback_frames,
         labeled_marker_names = labeled_marker_names, 
         check_hand_skeleton_heuristics = check_hand_skeleton_heuristics,
         use_skeleton = use_skeleton,
         debug =debug,
         ignore_marker_names = ignore_marker_names,
         plot_every_X_frames = plot_every_X_frames
        )

Further functionalities

The Take object can then be used to plot any given frame for inspection of the labeled data, by using
plotAt(frame, filename="")
By default the plot is displayed and not saved. filename = "" Defines the filename for saving the plot. If given, the plot is not shown to the user but only saved. The lower and upper limits of the x,y, and z axis can be quickly changed by setting the PLOT_X_LIM, PLOT_Y_LIM, PLOT_Z_LIM properties of the Take object.

%matplotlib inline

t.PLOT_X_LIM = (0.0,0.5)
t.PLOT_Y_LIM = (-0.2,0.2)
t.PLOT_Z_LIM = (-0.4,0.2)

t.plotAt(2219)

annafeit / motion-capture-labeling