moukle / BrickScanner

Table of Contents

1. About The Project
   • Built With
2. Getting Started
   • Prerequisites
   • Installation
3. Brick Identification Pipeline
   • Simulation-based Image Generation
   • 3D Reconstruction
   • Point Cloud Alignment and Rating
4. 3D Reconstruction from real Image Pairs
5. Color Detection
6. License

About The Project

This project contains the source code used in my master thesis written at the Technical University of Darmstadt in 2022.

Title: Automatic detection and sorting by using optical methods and enhanced computer vision algorithms on the example of LEGO® Bricks - Part: Scanning, 3D-Reconstruction & Model Matching

Abstract: In cooperation, the Institute of Materials Science (If W) and the Department of Graphical Interactive Systems (GRIS) initiated the project to implement a modern LEGO ® sorting machine. Due to the scope of this project, the realization of the machine was divided into three works.

In this work, a novel 3D reconstruction pipeline is introduced, combining two stereo camera setups with line lasers, for geometry and color independent shape retrieval. This setup allows reconstructing objects on continuous conveyor belt systems in real time. In addition, an OpenGL simulator is presented, which provides virtual image rendering for the introduced setup and builds the transition between OpenGL and OpenCV conventions. With that, the reconstruction pipeline, implemented and tested in simulation, is directly applicable to the real setup.

Furthermore, a classification for reconstructed point clouds is performed to categorize each reconstructed brick to one of the models from a database with approximately 5,000 models. Therefore, a bounding box similarity search, based on Principal Component Analysis (PCA), is performed, which reduces the amount of models for point cloud registration and alignment quality measure.

Built With

(back to top)

Getting Started

Recommended: You can use this repository. It conveniently packages all the necessary libraries within a Docker Image that is ready for immediate use.

Alternatively, you can follow the steps outlined below if you prefer not to use the provided Docker Image.

Prerequisites

This project uses the detailed brick models, provided by the LDraw community. In order to use this library the following steps are required.

1. Download and unpack the complete LDraw Parts Library

2. Install LDView, a real-time 3D viewer for displaying LDraw models. Its command line backend allows converting the LDraw models to the STL file format. After installation, check if the application is executable from command line via

   LDView

You can use this script to install LDView, LDraw and dependencies on Void Linux.

Installation

1. Clone the repo

   git clone https://github.com/flowmeadow/BrickScanner.git

2. Install the required pip packages

   pip install -r requirements.txt

3. Open the file definitions.py and update the paths IMG_DIR, DATA_DIR, BRICK_DIR and STL_DIR

   • Set BRICK_DIR to the current location of the 'parts' folder, which is inside the extracted LDraw parts library
   • Create and reference empty folders for IMG_DIR, DATA_DIR and STL_DIR

   # paths
   ROOT_DIR = os.path.dirname(os.path.abspath(__file__))
   IMG_DIR = os.path.join(ROOT_DIR, '_images')
   DATA_DIR = os.path.join(ROOT_DIR, '_data')
   BRICK_DIR = '/home/user/ldraw/parts'
   STL_DIR = os.path.join(ROOT_DIR, '_stl')

4. Convert models to the STL format with

   python scripts/create_stl_files.py

   This will start the conversion of each brick. Each model from BRICK_DIR is opened with the LDView application, converted to the STL format and stored in STL_DIR. This progress can take some time and requires approximately 2GB of memory.

Poetry

1. Install poetry with

   curl -sSL https://install.python-poetry.org | python3 -

2. Install python3.8.15 with

   curl https://pyenv.run | bash
   pyenv install -v 3.8.15
   
   poetry init
   poetry env use python3.8

3. Create STL files

   poetry install
   poetry self add 'poethepoet[poetry_plugin]'
   poetry poe stl

(back to top)

Brick Identification Pipeline

In this section, the simulation-based brick identification pipeline is introduced. An example for all the following steps is given in demo.py.

Simulation-based Image Generation

To evaluate the proposed pipeline, an OpenGL simulator is used to render stereo image pairs while a virtual brick is forwarded through the laser blade, realized with a shader.

Screenshot of the OpenGL simulator

Rendered images from the left and right camera view

3D Reconstruction

To obtain a 3D point cloud of the model, first, the section in the image pairs containing the laser light projection is isolated. Afterwards, using the epipolar constraints of the stereo camera views, corresponding 2D points from each two images are determined. These point pairs are used to compute the respective 3D points.

Simulation-based reconstructed point clouds for exemplary bricks

Point Cloud Alignment and Rating

Using Principal Component Analysis (PCA) and the Iterative Closest Point (ICP) algorithm, the most similar brick models from the LDraw database are selected and the optimal alignment is determined. Using the final alignment error of these bricks gives an estimate for the brick ID.

Demonstration of the different alignment steps

3D Reconstruction from real Image Pairs

./scripts/stereo_calibration.py
./scripts/world_space_calibration.py
./scripts/record_stereo_images.py

# unfinished script
./real_recon.py

(back to top)

Color Detection

WIP

(back to top)

License

Distributed under the MIT License. See LICENSE.rst for more information.

(back to top)