Real time point cloud processing and latency

Question

Our project is to integrate Lidar system into virtual reality (unity). I could achieve that integration with ROS-bridge. Next step is to process the point cloud data before we send it to unity system.

• Lidar sensor velodyne VLP-16
• Ubuntu 18.4
• IDE: Pycharm (python)
• Point cloud processing: python point cloud library with ROS
• ros-bridge with unity system

Problem Without processing, there is only 1 second latency from sensor to unity visualization. But processing point-cloud data in ROS(pycharm) causes significant latency (around 5 seconds).

• I am using velodyne drives to convert raw data into pointcloud2 format. In the code for the processing, I subscribe this pointcloud2 message and convert it into PointXYZRGB format to apply pcl libraries.
• I first tested this sequence of data-type conversion without applying PCL.

raw data -> publish pointcloud2 message ->subscribe pointcloud2- > pointXYZRGB -> (processing) -> pointcloud2 message -> publish it.

• Without processing, I get over 5 seconds latency.

I understand that it would be better to receive raw data from sensor without converting into pointcloud2 messages. But it is very challenging for me to do it in python. I found one grabber example in c++. https://medium.com/@yohei.kajiwara/vlp16-c-quick-example-35b9ceea2059

But I am not sure what is the most reliable way. Please give me an advice on this issue.

Best regards

Seongsu Byeon

Answer 1

If speed is your concern, there's a few things.

Velodyne drivers have the ability to receive the packets & combine into a pointcloud with zerocopy access, using nodelets. ROS nodelets are like nodes, but with less copying, more efficient for big data. Ex velodyne_driver and Ex velodyne_pointcloud. You can call it all through the launch file VLP16_points.launch which demonstrates how to combine nodelets like this. The code is pretty readable, so you could walk through how they setup their code for reference. If you haven't tried this method, and then piping into Unity, this is your minimum baseline. Then the bottleneck is on the unity side.

The fastest you'll get is probably writing a nodelet in C++, using the PCL, handled by the same nodelet manager (as in that last launch file). The best is if you don't need to write any yourself. The pcl_ros package has several transform nodelets; if a combination of those could meet your needs, that would be the most efficient & effortless way. Otherwise, you can still use the <pcl_ros/pointcloud.h> header in your c++ nodelet, and the pcl::PointCloud<T> will be interpreted by ros as a sensor_msgs/PointCloud2 message on both subscriber and publisher. If you need anymore conversions, the pcl_conversion package has a few handy ones.

(This is a bit tangental. If the bottle neck beyond that is the websocket-y interface Ros# uses to communicate with the rosbridge_server, rosbridge_server supports a UDP protocol, which, if on the same machine, should be pretty quick, but Ros# doesn't currently support that or have it on their roadmap, as most of their use-case doesn't come from it running on the same machine.)