The Mid-level Vision Toolbox (MLVToolbox) was created by the BWLab at the University of Toronto as a tool for researchers to extract structural properties of contours such as orientation, length, curvature and junctions present in images. MLVToolbox allows researchers to quantify the presence of perceptual organization cues such as mirror symmetry, ribbon symmetry, and taper symmetry using medial axis properties of contours in naturalistic images. MLVToolbox also provides researchers with a free and easy way to extract visual statistics that underlie our experience of surfaces, objects, scenes and even works of art. It will facilitate reproducible experimental stimulus sets using built-in functions that manipulate contours in images based on one or several mid-level vision properties. In addition, MLVToolbox offers several visualization functions for tracing contour orientations, lengths, curvature, junctions, as well as, medial axis properties on computer-generated and artist-generated line drawings. A contour extraction functionality helps users obtain computer-generated line drawings from real-world photographs. To encourage researcher's exploration of its potential, we have provided several example datasets produced using the MLVToolbox.
We are working on the python version and we have a plan to release the python version soon.
Please stay tuned for updates!
- Matlab
- Matlab Computer Vision Toolbox
- Matlab Image Processing Toolbox
- Matlab Statistics and Machine Learning Toolbox
Setup Path
The first step to use the MLV toolbox is to add the required folders to the Matlab path. You can do this on Matlab's Graphical User Interface or by running the following line of code in the command window:
setupExtracting Line Drawings
The main function for extracting line drawings is "lineDrawingTracing.m".
Please note that the line drawing extraction process can take a while to finish, depending on the image size.
Example usage:
fileName= 'example.jpg';
vecLD = lineDrawingTracing(fileName);
figure;
subplot(1,2,1);
imshow(imread('example.jpg'));
subplot(1,2,2);
drawLinedrawing(vecLD,3,'b');
Output:
Note that the drawLineDrawing takes a LineDrawing (LD) data structure (represented as vecLD in the example above), and draws the outcome with specific line width (3 in the example above) and a chosen color ('b' -> which is for blue in Matlab).
For a runnable matlab live code (.mlx), please refer to the script.
Computing Medial Axis Properties
Using MLV, one can compute accurate AOF-based medial axis transform (MAT) from binary images. These images can either be rendered from LineDrawing (LD) datastructures (check out renderLinedrawing.m function) or be binary images from other sources. Like the example (bunny) above shows, there are some intermediate steps in the process of extracting MAT, including the extraction of the distance map as well as the extraction of the average outward flux map (AOF). This information along with the skeleton is stored in the MAT computed from a binary image. Please see the following example of a mountain scene where the MAT is computed from the binary image.
img = imread('images/mountain.png');
MAT = computeMAT(img,28);
figure;
subplot(1,2,1);
imshow(img);
subplot(1,2,2);
imshow(imoverlay(rgb2gray(img),MAT.skeleton,'b'))
Output:
In this example, 28 represents the object angle degree that the medial axis is thresholded at. For further information, please refer to the book: Medial Representations. The higher the threshold, the less of medial axis branches are retained in the final representation.
The final MAT includes the following attributes in its data structure:
- distance_map (step b)
- AOF (step c)
- skeleton (step d)
For a runnable Matlab live code (.mlx) with more examples to run, please refer to the script.
Computing Contour Properties
Using the MLV toolbox, there are two major types of contour properties that one can compute:
- Medial-axis based properties
- Contour based properties
To compute medial-axis based properties, the user should first compute MAT representation first and then use the proper needed functions. Take a look at the following block of code as an example.
load('dataSets/TorontoScenes/mountains_vecLD.mat');
cute = vecLD(11);
imgLD = renderLinedrawing(cute);
MAT = computeMAT(imgLD,28);
[MATcontourImages,MATskeletonImages,skeletalBranches]=computeAllMATproperties(MAT,imgLD);
figure;
subplot(1,2,1);
drawMATproperty(MATskeletonImages.mirror);
title('Mirror Symmetry','FontSize',24);
subplot(1,2,2);
drawMATproperty(MATcontourImages.mirror);
title('Mirror Symmetry','FontSize',24);
Output:
As shown above, we see the local mirror symmetry scores on an example LineDrawing (LD) structure, where colors show low to high scores from blue to red respectively. Check out a more detailed example Matlab Live Script (.mlv) here.
Similar to medial-axis based properties, one can also compute contour based properties using the MLV toolbox. Please take a look at the following block of code as an example.
load('dataSets/TorontoScenes/mountains_vecLD.mat');
cute = vecLD(11);
cute = computeContourProperties(cute);
figure;
subplot(1,2,1);
drawLinedrawingProperty(cute,['orientation']);
subplot(1,2,2);
drawLinedrawingProperty(cute,'junctions');
Output:
Again, check out a more detailed example Matlab Live Script (.mlv) here.
Manipulating Line Drawings
We are still working on this part of the documentation ...
Batch Processing
Will be available in future releases ...
In this toolbox, we introduce the TorontoScene artist drawn dataset. Color photographs of six categories of natural scenes (beaches, city streets, forests, highways, mountains, and offices) were downloaded from the internet, and those rated as the best exemplars of their respective categories by workers on Amazon Mechanical Turk were selected. Line drawings of these photographs were generated by trained artists at the Lotus Hill Research Institute. Artists traced the most important and salient lines in the photographs on a graphics tablet using a custom graphical user interface. Contours were saved as successions of anchor points. Photogaphs and line drawing images are available here. These scenes were used for many papers from our lab.
The International Affective Picture System (IAPS) provides normative ratings of emotion (pleasure, arousal, dominance) for a set of color photographs that provide a set of normative emotional stimuli for experimental investigations of emotion and attention. The images are freely available upon request. Line drawings of the IAPS images were generated by trained artists at the Lotus Hill Research Insititute with the same instructions as the TorontoScenes images. These line drawings were used in this study:
Claudia Damiano, Dirk B. Walther, & William A Cunningham, Contour features predict valence and threat judgements in scenes. Sci Rep 11, 19405 (2021). https://doi.org/10.1038/s41598-021-99044-y
We are working on this section, please make sure to check back here in the future. As of now, please ask your questions on the discussion page of the MLV toolbox available here.
If you use the MLV toolbox, we appreciate it if you cite the following papers:
@InProceedings{rezanejad2019scene,
author = {Rezanejad, Morteza and Downs, Gabriel and Wilder, John and Walther, Dirk B and Jepson, Allan and Dickinson, Sven and Siddiqi, Kaleem},
title = {Scene Categorization from Contours: Medial Axis Based Salience Measures},
booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2019}
}
@article{walther2014nonaccidental,
title={Nonaccidental properties underlie human categorization of complex natural scenes},
author={Walther, Dirk B and Shen, Dandan},
journal={Psychological science},
volume={25},
number={4},
pages={851--860},
year={2014},
publisher={Sage Publications Sage CA: Los Angeles, CA}
}
@incollection{rezanejad2013flux,
title={Flux graphs for 2D shape analysis},
author={Rezanejad, Morteza and Siddiqi, Kaleem},
booktitle={Shape Perception in Human and Computer Vision},
pages={41--54},
year={2013},
publisher={Springer}
}
The MLV toolbox contains various modules from different projects. Please take some time to check licenses on the released files.