For a full tutorial writeup: https://docs.edgeimpulse.com/experts/ros2-part2-microros
Click image for YouTube tutorial video:
- Arduino Portenta H7 + vision shield (more boards coming soon!)
- Linux computer running ROS2
You'll need to install a few things in order to follow along with this tutorial:
Clone the library from this repository and add the .ZIP folder to your Arduino IDE. This library comes precompiled, but we'll need to rebuild it after we add the custom Edge Impulse ROS2 message types (to be discussed).
To ease the process of interfacing Edge Impulse with micro-ROS two custom message types were created:
EIClassification: Contains a label and value, like{'label': 'cat', 'value': 0.75}. One classification contains one class name and the probability given to that class by the neural network.EIResult: Contains multiple classifications - as many as your neural network needs. A full result looks like this:[{'label': 'cat', 'value': 0.75}, {'label': 'dog', 'value': 0.25}].
In order to use these message types they need to be added to both your ROS2 and micro-ROS environments. Clone the micro-ROS + Edge Impulse repository here and copy the ei_interfaces directory. This folder contains everything you need to build the custom message types.
To add it to your ROS2 system, navigate to:
ros2_ws/src
and paste the ei_interfaces directory inside. cd back to your main ros2_ws directory and from the terminal run colcon build.
You can confirm the message types were added by running the following from the terminal:
ros2 interface list | grep EI
You should see:
ros2 interface list | grep EI
ei_interfaces/msg/EIClassification
ei_interfaces/msg/EIResult
To add it to your MicroROS environment, navigate to the micro-ROS Arduino library (that you cloned added to the Arduino IDE). You need to paste the same ei_interfaces directory inside the special extra_packages directory in the Arduino library. For me the path is:
~/Arduino/libraries/micro_ros_arduino-2.0.5-humble/extras/library_generation/extra_packages
Paste the directory there, return to the main micro_ros_arduino-2.0.5-humble directory, and use the docker commands from this part of the MicroROS Arduino readme:
Note: You may need to make the
library_generation.sh(in/extras/library_generation) able to run as an executable (right click and change permissions) in order for the library builder to work!
docker pull microros/micro_ros_static_library_builder:humble
docker run -it --rm -v $(pwd):/project --env MICROROS_LIBRARY_FOLDER=extras microros/micro_ros_static_library_builder:humble -p portenta-m7
Note the -p flag at the end - it significantly reduces the build time if you specify your target. You can also run the command without this flag to build for all available targets, but it'll take a while.
Now it's time to export your Edge Impulse vision project as an Arduino library, and be sure to add the .ZIP folder to the Arduino IDE.
As for the example code for this project, find it here. Compile and upload the .ino file to your Arduino Portenta, and make sure the .h header file is in the same directory. I won't be writing a line-by-line explanation of the code here - but here is some info on key points that make this all work.
Make sure to change the name of the included Edge Impulse library to the name of your own project:
// Replace this with <name_of_your_ei_library_inferencing.h>
#include <micro_ros_ei_inferencing.h>Inside the ei_result_publisher file, note that we include the two message types we added before:
#include <ei_interfaces/msg/ei_result.h>
#include <ei_interfaces/msg/ei_classification.h>The reason we need to add both is because EIResult is a sequence (array) of EIClassification messages, and in MicroROS you need to allocate memory for your message when setting everything up. Even if your neural network has more labels than than the 2 that I have for this project (human, background), the code will still work fine as it will automatically allocate enough memory for however many labels (and hence classifications) your EIResult message needs to support. You can see the section where the memory is allocated here:
msg.result.capacity = LABEL_COUNT;
msg.result.data = (ei_interfaces__msg__EIClassification*) malloc(msg.result.capacity * sizeof(ei_interfaces__msg__EIClassification));
msg.result.size = 0;
// Allocate memory to message
for (int32_t ix = 0; ix < LABEL_COUNT; ix++) {
// If 20 characters isn't enough - increase this value
msg.result.data[ix].label.capacity = 20;
msg.result.data[ix].label.data = (char*) malloc(msg.result.data[ix].label.capacity * sizeof(char));
msg.result.data[ix].label.size = 0;
msg.result.size++;
}Note that our msg is initialized as type:
ei_interfaces__msg__EIResult msg;You can see the names of the node and publisher:
RCCHECK(rclc_node_init_default(&node, "ei_micro_ros_node", "", &support));
...
RCCHECK(rclc_publisher_init_default(
&publisher,
&node,
ROSIDL_GET_MSG_TYPE_SUPPORT(ei_interfaces, msg, EIResult),
"/ei_micro_ros_publisher"));These names are what will appear on your ROS2 system once the microo-ROS agent detects your micro-ROS publisher.
In the .ino file, you'll see that a lot of the code is taken directly from the Edge Impulse ei_camera example code here. Let's focus on the moment that the ei_impulse_result_t object is transferred to the MicroROS publisher:
// Run the classifier
ei_impulse_result_t result = { 0 }; // Initialize result
EI_IMPULSE_ERROR err = run_classifier(&signal, &result, debug_nn); // Run classifier
if (err != EI_IMPULSE_OK) {
return;
}
fill_result_msg(result); // Store result data in MicroROS message
publish_msg(); // Publish messageOK, now it's time to run the MicroROS agent and see if our node is publishing as expected. The agent runs on your main ROS2 computer and serves as a middle man to allow your micro-ROS device to communicate with your main ROS2 system. It's recommended to use the docker command for the agent. When you run this command be sure and use paste in your board port - in my case the Portenta H7 connects to /dev/ttyACM0.
docker run -it --rm -v /dev:/dev --privileged --net=host microros/micro-ros-agent:humble serial --dev [YOUR BOARD PORT] -v6
Since you'll probably be using this command a bunch, you might find it convenient to make an alias for it :)
After starting the agent, you may have to reset your Arduino (with the reset button, or just unplug and reconnect).
In a separate terminal, check if the topic is listed. You should see the name of your topic:
ros2 topic list
/ei_micro_ros_publisher
...
To see the result messages, echo the topic:
ros2 topic echo /ei_micro_ros_publisher
And if everything worked you should see the result messages:
.
.
.
result:
- label: background
value: 0.75390625
- label: human
value: 0.24609375
---
result:
- label: background
value: 0.69140625
- label: human
value: 0.30859375
---
result:
- label: background
value: 0.71875
- label: human
value: 0.28125
---
.
.
.
Now you can subscribe to this topic as you would any other ROS2 topic! You can access the labels and values like this:
# Remember that the result is an array of classifications
msg.result[i].label # 'dog'
msg.result[i].value # 0.75