rsyslog inside docker containers => "rsyslogd is not running ... failed"

Question

I am running rsyslog within docker containers to send UDP messages to logstash.

When I log into the docker container, and type:

service rsyslog status

shows:

rsyslogd is not running ... failed! 

However, while I am in the container, if I type:

service rsyslog start 

It starts up perfectly with no errors and no real sign of why it failed at the start

I CAN NOT FIGURE OUT WHY IT IS FAILING!!!!

*The rsyslog conf file has not been modified except the Modules to allow for imfile. The rsyslog.conf is as follows:

#  /etc/rsyslog.conf    Configuration file for rsyslog.
#
#                       For more information see
#                       /usr/share/doc/rsyslog-doc/html/rsyslog_conf.html
#################
#### MODULES ####
#################
module(load="imfile" PollingInterval="10")
module(load="imuxsock" )  # provides support for local system logging
module(load="immark")  #provides --MARK-- message capability

###########################
#### GLOBAL DIRECTIVES ####
###########################

#
# Use traditional timestamp format.
# To enable high precision timestamps, comment out the following line.
#
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

#
# Set the default permissions for all log files.
#
$FileOwner root
$FileGroup adm
$FileCreateMode 0644
$DirCreateMode 0755
$Umask 0022

#
# Where to place spool and state files
#
$WorkDirectory /var/spool/rsyslog

#
# Include all config files in /etc/rsyslog.d/
#
$IncludeConfig /etc/rsyslog.d/*.conf


###############
#### RULES ####
###############

#
# First some standard log files.  Log by facility.
#
auth,authpriv.*                 /var/log/auth.log
*.*;auth,authpriv.none          -/var/log/syslog
#cron.*                         /var/log/cron.log
daemon.*                        -/var/log/daemon.log
kern.*                          -/var/log/kern.log
lpr.*                           -/var/log/lpr.log
mail.*                          -/var/log/mail.log
user.*                          -/var/log/user.log

#
# Logging for the mail system.  Split it up so that
# it is easy to write scripts to parse these files.
#
mail.info                       -/var/log/mail.info
mail.warn                       -/var/log/mail.warn
mail.err                        /var/log/mail.err

#
# Logging for INN news system.
#
news.crit                       /var/log/news/news.crit
news.err                        /var/log/news/news.err
news.notice                     -/var/log/news/news.notice

#
# Some "catch-all" log files.
#
*.=debug;\
        auth,authpriv.none;\
        news.none;mail.none     -/var/log/debug
*.=info;*.=notice;*.=warn;\
        auth,authpriv.none;\
        cron,daemon.none;\
        mail,news.none          -/var/log/messages

#
# Emergencies are sent to everybody logged in.
#
*.emerg                         :omusrmsg:*

#
# I like to have messages displayed on the console, but only on a virtual
# console I usually leave idle.
#
#daemon,mail.*;\
#       news.=crit;news.=err;news.=notice;\
#       *.=debug;*.=info;\
#       *.=notice;*.=warn       /dev/tty8

# The named pipe /dev/xconsole is for the `xconsole' utility.  To use it,
# you must invoke `xconsole' with the `-file' option:
#
#    $ xconsole -file /dev/xconsole [...]
#
# NOTE: adjust the list below, or you'll go crazy if you have a reasonably
#      busy site..
#
daemon.*;mail.*;\
        news.err;\
        *.=debug;*.=info;\
        *.=notice;*.=warn       |/dev/xconsole

*I have a script file that starts rsyslog

if [[ -z "$(pgrep rsyslog)" ]]; then
  echo "starting rsyslog"
  service rsyslog start
fi

My conf file is as follows:

##Get Nginx Error Logs
$InputFileName /var/log/nginx/error.log
$InputFileTag http-error
$InputFileStateFile stat-nginx-error
$InputFileSeverity error
$InputFileFacility local7
$InputRunFileMonitor

#GRAB PHP-FPM ACCESS LOGS
$InputFileName /var/log/php-fpm/access_log
$InputFileTag php-fpm-access
$InputFileStateFile stat-php-fpm-access
$InputFileSeverity info
$InputFileFacility local7
$InputRunFileMonitor

#GRAB PHP-FPM ERROR LOGS
$InputFileName /var/log/php-fpm/error_log
$InputFileTag php-fpm-error
$InputFileStateFile stat-php-fpm-error
$InputFileSeverity error
$InputFileFacility local7
$InputRunFileMonitor

#Json Template

template(name="json_temp" type="list")
  { constant(value="{")
    constant(value="\"@timestamp\":\"")         property(name="timegenerated" dateFormat="rfc3339")
    constant(value="\",\"message\":\"")         property(name="msg")
    constant(value="\",\"severity_label\":\"")  property(name="syslogseverity-text")
    constant(value="\",\"severity\":\"")        property(name="syslogseverity")
    constant(value="\",\"facility_label\":\"")  property(name="syslogfacility-text")
    constant(value="\",\"facility\":\"")        property(name="syslogfacility")
    constant(value="\",\"program\":\"")         property(name="programname")
    constant(value="\",\"pid\":\"")             property(name="procid")
    constant(value="\",\"rawmsg\":\"")          property(name="rawmsg")
    constant(value="\",\"syslogtag\":\"")       property(name="syslogtag")
    constant(value="\"}\n")
  }

if $programname == 'http-error' then @ip.address:port;json_temp
if $programname == 'http-error' then stop
if $programname == 'php-fpm-access' then @ip.address:port;json_temp
if $programname == 'php-fpm-access' then stop
if $programname == 'php-fpm-error' then @ip.address:port;json_temp
if $programname == 'php-fpm-error' then stop


*.* @ip.address:port;json_temp

Any help would be awesome because I do not understand why it is not starting up.

Cheers

Answer 1

This is my rsyslog.conf in docker container(centos7):

$> cat /etc/rsyslog.conf |grep -vE '^$|^#'
$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$WorkDirectory /var/lib/rsyslog
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
$IncludeConfig /etc/rsyslog.d/*.conf
$OmitLocalLogging off
*.info;mail.none;authpriv.none;cron.none                /var/log/messages
authpriv.*                                              /var/log/secure
mail.*                                                  -/var/log/maillog
cron.*                                                  /var/log/cron
*.emerg                                                 :omusrmsg:*
uucp,news.crit                                          /var/log/spooler
local7.*                                                /var/log/boot.log

Also i changed /etc/rsyslog.d/listen.conf

#$SystemLogSocketName /run/systemd/journal/syslog

And then

$> rsyslogd -n 

Thanks @elinax

More info at https://www.projectatomic.io/blog/2014/09/running-syslog-within-a-docker-container/

Answer 2

We bumped upon the same issue on a Docker 17.03.2-ce image created on CentOS 7.3.1611. The solution is in verifying /etc/rsyslog.conf as per this documentation. Basically, in /etc/rsyslog.conf:

• Remove $ModLoad imjournal
• Set $OmitLocalLogging to off
• Make sure $ModLoad imuxsock is present
• Comment out: $IMJournalStateFile imjournal.state

Finally, note that running the rsyslogd or anything else, is the responsibility of the program that is being run inside the container. It is not going to get launched, automatically.