How can I redirect tail -f output to curl (HTTP stream)

Question

I want to collect realtime logs from embedded devices. The device have limited command. It would be great if I can do it by a shell script.

So, can I do something like this tail -f /var/logs/some-log.log | curl -X POST --data-urlencode @- http://<log-collaction.website>?

I have tried many ways, but all failed. For example:

ping www.google.com | while read -r LINE; do curl -X POST --silent --data-binary $LINE "http://localhost:8080/"; done

The above command works, but it will post messages line by line. However, I think the performance is not good if I have a lot of content to send.

I know HTTP/1.1 support chunked encoding. How can I redirect a program's stdout to curl or a HTTP stream (maybe long connection?).

Thanks

Answer 1

You can use a tool like buffer (or perhaps even better mbuffer) to combine both strategies.

tail -f  /var/logs/some-log.log | buffer | while read -r LINE; do curl -X POST --silent --data-binary $LINE "http://localhost:8080/"; done

This reduces the number of HTTP requests by curl to your webservice (but possibly makes you lose some logs when your connection dies).

However, I agree with Nick Russo that implementing websockets might be a better idea altogether.

Hope this helps!