Baler systemd service

Intro

balerd, like other daemons, can be manually invoked (please see man /opt/ovis/share/man/man1/balerd.1). However, for convenience, we also provide a systemd script so that balerd can be easily start and stop through sytemctl interface. Another benefit to use systemd is that the daemon logs are managed by system logging mechanism, and we don’t have to worry about the log file.

For more information about baler, please see here.

Configure and Control

There are two configuration files that control systemd-baler daemon: /opt/ovis/etc/baler/balerd.master.env and /opt/ovis/etc/baler/balerd.master.conf. The env file controls command-line options and environment variables, while the conf file controls plugin-related configurations. Please see  each option explanation in the following example files:

Examples:

# file: /opt/ovis/etc/baler/balerd.master.env

# This file contain environment variables that affect how balerd.master.service
# starts baler daemon.

# store path, change it to your liking (e.g. `/NVME/0/master.store`)
BALERD_STORE=/opt/ovis/var/lib/baler/master.store

# Plugin configuration file
BALERD_PLUGIN_CONFIG=/opt/ovis/etc/baler/balerd.master.conf

# master port (to serve other slave balerd)
BALERD_MASTER_PORT=10003

# Log level
BALERD_LOG_LEVEL=WARN

# The number of input queue workers
BALERD_IN_Q_WORKERS=1

# The number of output queue workers
BALERD_OUT_Q_WORKERS=1

# The following options are configured by automake, please don't change them.
ZAP_LIBPATH=/opt/ovis/lib64/ovis-lib
# file: /opt/ovis/etc/baler/balerd.master.conf

# baler depends on English word list to form a pattern. The default word list should
# contain enough words.
tokens type=ENG path=/opt/ovis/etc/baler/eng-dictionary
# It is also OK to add more than one word list.
tokens type=ENG path=/some/other/word/list

# baler also depends on the host list to recognize hostnames
tokens type=HOST path=/opt/ovis/etc/baler/hosts.txt
# Like word list, it is also OK to have multiple host list
tokens type=HOST path=/other/hosts/list

# Image output configuration
# In this example, we will generate minute (60 sec) and hour (3600 sec) image data.
plugin name=bout_sos_img delta_ts=60
plugin name=bout_sos_img delta_ts=3600

# Message output
plugin name=bout_sos_msg

# Input plugin to process live `rsyslog` messages, listening on port 10514 (mimicking
# rsyslog port 514).
plugin name=bin_rsyslog_tcp port=10514

Then, simply use systemctl commands to start, stop and check status of the daemon. For example:

systemctl start balerd.master
systemctl status balerd.master
systemctl stop balerd.master

Automatically Start the daemon at system startup

systemctl enable balerd.master is the systemctl command to enable balerd.master service to start automatically at boot. If the command doesn’t work due to systemctl bug (here), the work-around is to manually create the soft-link in /etc/systemd/system/multi-user.target.wants/ directory as follow:

ln -s /opt/ovis/etc/systemd/system/balerd.master.service \
      /etc/systemd/system/multi-user.target.wants/
systemctl daemon-reload 

After enabling it using the work-around, if you wish to disable it (i.e. not start automatically at boot), systemctl disable balerd.master will work just fine.

Rsyslog forwarding, SELinux, and Firewall

Before configuring rsyslog to forward the log messages to baler, if we have firewall and/or Security-Enhanced Linux enabled, we have to add the port used in bin_rsyslog_tcp above to the security services.

# get the `semanage` utility to easily manage SE Linux
$ yum install policycoreutils-python

# If SELinux is enabled, we have to add port 10514/tcp, and associating it 
# with syslogd_port_t type.
$ selinuxenabled && semanage port -a -t syslogd_port_t -p tcp 10514

# If the firewalld.service is in effect, we also need to allow 10514 in the firewalld 
$ firewall-cmd --add-port 10514/tcp
$ firewall-cmd --add-port 10514/tcp --permanent
# firewall-cmd is quite strange ... `--permanent` option makes it permanent, but not
# effective immediately (see firewall-cmd(1) man page). That's why we need to call
# the command twice, with and without --permanent option.

This needs to be done on both the system hosting balerd and other systems forwarding logs to it.

The following is an example of setting up rsyslog to forward log messages to a baler daemon.

# Add the forwarding rule at the end of /etc/rsyslog.conf.
# The following is an example output of the tail of rsyslong.conf
$ tail -n5 /etc/rsyslog.conf 
#$ActionResumeRetryCount -1 # infinite retries if host is down 
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
#*.* @@remote-host:514 
*.* @@192.168.56.22:10514 
# ### end of the forwarding rule ###

# NOTE1: The rule forward all messages over TCP to 192.168.56.22 port 10514
# NOTE2: If using a hostname in the `remote-host` doesn't work, try using an IP address
#       instead.

# Then, restart the rsyslog service
$ systemctl restart rsyslog.service  

Investigating the Daemon Log

The daemon log will go to system logging facility. Use the following command to see balerd.master daemon logs.

$ journalctl _SYSTEMD_UNIT=balerd.master.service

HPC Machine Data Mining