ROS: Is ZeroMQ better for large data streams, e.g. raw images, than native image topic?

Question

Fairly new to ROS, but haven't been able to find this information searching around.

We're building an instrument where we need to transfer large data streams over the network on a 100Mbit limited cable. Preferably we need to transfer RAW images (~10MB a piece) or we can do some lossless compression resulting in about 5MB a piece.

Is this perfectly fine for ROS with native image topics, or would a separate protocol such as ZeroMQ be better for this task? What are the limitations in ROS on large data streams?

Hope that someone with the knowledge could take a second to share some experience.

Thanks!

Answer 1

10E6 [B] over a private, 100% free 100E6 [b/s] channel takes no less ~0.8 [s]
_5E6 [B] over a private, 100% free 100E6 [b/s] channel takes no less ~0.4 [s]

Q : What are the limitations in <something> on large data streams?

Here we always fight a three-fold Devil mix of:

Power( for data processing, a 10[MB]->5[MB] compression + RAM-I/O not being excluded )
+
Time( latency + jitter of the E2E data-transport across the series of transport-channels )
+
Errors( uncertainties of content delivery, completeness and authenticity over the E2E data-transport path )

In the ROS domain, being a system for coordinated control-loops' constrained sub-systems, there is one more problem - not meeting the "in-time-ness" causes the control to fail into principally unstable territory.

Countless examples of what happens when this border has been crossed - from production line falling into panic, resulting in an immediate emergency total-stop state, to damaged tools, products, equipment and continued colliding, still crashing during still continued operations ( when collision-detection and emergency total-stops were not implemented safe ).

---

Q : would a separate protocol such as ZeroMQ be better for this task?

ZeroMQ has excellent performance ( does not add much on the Time leg of the Devil-mix, yet it always depends on having (in-)sufficient resources ( the Power to smoothly process )

ZeroMQ has excellent performance scalability, sure, if the Power leg of the Devil-mix permits.

ZeroMQ has excellent properties for the Errors leg of the Devil-mix - we get warranty of Zero-Errors - it either delivers the message (the payload) as a bit-to-bit identical copy of the original content, or nothing at all. This warranty may look strange, a sure overkill for blurred or noisy images - yet, it is a fair strategy for not having additional Power and Time-uncertainty issues due to error-detection/limited-recovery. Yet it leaves us free with choices, how to handle (if needed), within a given, constrained, Time- and Power-domains, the main duty - the ROS control-loops' stability - with missing or re-transmit requested payloads, given Errors were indirectly detected from time-stamping or monotonic-ordinal indexing et al.

ROS Topics, on the contrary, are limited to a single PUB/SUB formal communication-pattern archetype only and fixed to use either a TCPROS transport-class ( ZeroMQ may use a faster, L3+protocol stack-less { inproc:// | ipc:// } or, if needed, even extended to a mil-std guaranteed-delivery or a distributed grid-computing tipc:// or hypervisor-orchestrated vmci:// transports ) or UDPROS, which is currently available in roscpp only and lossy, but having lower latency, compared to TCPROS.