I have a question, and I could'nt find help anywhere on stackoverflow or the web. I have a program (celery distributed task queue) and I have multiple instances (workers) each having a logfile (celery_worker1.log, celery_worker2.log). The important errors are stored to a database, but I like to tail these logs from time to time when running new operations to make sure everything is ok (the loglevel is lower). My problem: these logs are taking a lot of disk space. What I would like to do: be able to "watch" the logs (tail -f) only when I need it, without them taking a lot of space. My ideas until now: <ul> <li>outputing logs to stdout, not to a file: not possible here since I have many workers outputing to different files, but I want to tail them all at once (tail -f celery_worker*.log)</li> <li>using logrotate: it is an "OK" solution for me. I don't want this to be a daily task but would rather not put a minute crontab for this, and more, the server is not mine so that would mean some work on the admin-sys side</li> <li>using named pipes: it looked good at first sight but I didn't know that named pipes (linux FIFO) where blocking. Hence, when I don't tail -f ALL of the pipes at the same time, or when I just quit my tail, the writing operations from the logger are blocked.</li> </ul> Is there a way to have a non-blocking named pipe, which would just throw to stdout when tailed, and throw to /dev/null when not? Or are there technical difficulties to such a type of pipe? If there are, what are they? Thank you for your answers!

Have each worker log to stdout, but connect each stdout to a utility that automatically spools and rotates logs based on size or time. multilog and svlogd are examples of such. For those programs, you'd merely tail the "current" log file. You're right that logrotate is not quite the right solution for the problem you have. Named pipes won't work as you want. At best, your writers could fill up their pipes and then discard subsequent logs, which is the inverse of the behavior you want.

Logging to a non blocking named pipe?

I have a question, and I could'nt find help anywhere on stackoverflow or the web.

I have a program (celery distributed task queue) and I have multiple instances (workers) each having a logfile (celery_worker1.log, celery_worker2.log).

The important errors are stored to a database, but I like to tail these logs from time to time when running new operations to make sure everything is ok (the loglevel is lower).

My problem: these logs are taking a lot of disk space. What I would like to do: be able to "watch" the logs (tail -f) only when I need it, without them taking a lot of space.

My ideas until now:

outputing logs to stdout, not to a file: not possible here since I have many workers outputing to different files, but I want to tail them all at once (tail -f celery_worker*.log)
using logrotate: it is an "OK" solution for me. I don't want this to be a daily task but would rather not put a minute crontab for this, and more, the server is not mine so that would mean some work on the admin-sys side
using named pipes: it looked good at first sight but I didn't know that named pipes (linux FIFO) where blocking. Hence, when I don't tail -f ALL of the pipes at the same time, or when I just quit my tail, the writing operations from the logger are blocked.

Is there a way to have a non-blocking named pipe, which would just throw to stdout when tailed, and throw to /dev/null when not?

Or are there technical difficulties to such a type of pipe? If there are, what are they?

Thank you for your answers!

Are Named Pipes blocking?

Both reading and writing are by default blocking. This means that if a process tries to read from a named-pipe that does not have data, it will block. Similarly, if a process write into a named-pipe that has not yet been opened by another process, the writer will block. 3.

Can Named Pipes be bidirectional?

The named pipes feature enables bidirectional data or message exchange between applications that are on the same machine or applications that are on separate machines across a network.

What is the difference between a named and unnamed pipe?

An unnamed pipe is a direct connection between two commands running in the same terminal. If we want to send output from a command in one terminal to another command in a different terminal, we can use a named pipe, or FIFO. FIFO stands for first in, first out. This is a pipe that exists in the file system.

Are Named Pipes permanent?

A traditional pipe is "unnamed" and lasts only as long as the process. A named pipe, however, can last as long as the system is up, beyond the life of the process. It can be deleted if no longer used.

Have each worker log to stdout, but connect each stdout to a utility that automatically spools and rotates logs based on size or time. multilog and svlogd are examples of such. For those programs, you'd merely tail the "current" log file.

You're right that logrotate is not quite the right solution for the problem you have.

Named pipes won't work as you want. At best, your writers could fill up their pipes and then discard subsequent logs, which is the inverse of the behavior you want.

Logging to a non blocking named pipe?

Tags:

linux

bash

logging

pipe

fifo

Noé Malzieu

People also ask

1 Answers

pilcrow

Recent Activity

Donate For Us

Logging to a non blocking named pipe?

Tags:

linux

bash

logging

pipe

fifo

Noé Malzieu

People also ask

1 Answers

pilcrow

Related questions

Recent Activity

Donate For Us