There is an increasing interest in exploiting human pose estimation (HPE) software in human-machine interaction systems. Nevertheless, adopting such a computer vision application in real industrial scenarios is challenging. To overcome occlusion limitations, it requires multiple cameras, which in turn require multiple, distributed, and synchronized HPE software nodes running on resource-constrained edge devices. We address this challenge by presenting a real-time distributed 3D HPE platform, which consists of a set of 3D HPE software nodes on edge devices (i.e., one per camera) to redundantly extrapolate the human pose from different points of view. A centralized aggregator collects the pose information through a shared communication network and merges them, in real time, through a pipeline of filtering, clustering and association algorithms. It addresses network communication issues (e.g., delay and bandwidth variability) through a two-levels synchronization, and supports both single and multi-person pose estimation. We present the evaluation results with a real-case of study (i.e., HPE for human-machine interaction in an intelligent manufacturing line), in which the platform accuracy and scalability are compared with state of the art approaches and with a marker-based infra-red motion capture system.
This repository contains both the edge node code and the centralized aggregator code, respectively in edge_node_mqtt and aggregator. The first must be executed on edge devices, i.e. Nvidia Jetson, the second on a centralized server. The devices must be connected to the same communication network. A mqtt broker (e.g. mosquitto) is required on a device in the network for messages exchange.
The edge node requires:
- jetpack4.6.1
- trt_pose @ a89b422
- json @ db78ac1
- cmake >= 3.12.2
- torch == 1.7.0
- torchvision == 0.8.1
- paho mqtt == 1.3.8
- OpenCV
- Stereolabs Zed SDK
To install dependencies for the aggregator, on the centralized server:
cd BeFine/aggregator/
pip3 install -r requirements.txt
In addition edge nodes and the server aggregator need to be time synchronized with an NTP client (e.g. chrony) connected to a common NTP server (e.g. the server aggregator itself). In addition, the edge nodes take in input a configuration file with mqtt network configuration, topic, intrinsics and extrinsics matrix for common world reference. Finally, the server aggregator also need a configuration file with mqtt network configuration, input topics and aggregation settings.
On the server:
cd aggregator/
python3 multip_aggregator.py -c <conf_file>
On the edge node:
cd edge_node_mqtt/
mkdir build
cd build
cmake ..
make -j
./pub_kp3d -fconf <conf_file>