Linux Syslog Server
Server Administration

Linux Syslog Server and Log Management

In this post, we will talk about Linux Syslog Server and how to manage your logs, since Syslog format being the closest thing to the universal logging standard that exists.

If you want your system to be secured and hardened, you have to know what’s going on in that system, you can do that using logs. With logs, you can diagnose problems and determine the status of your system and applications.

In a previous post, we talked about how to secure Linux server and we mentioned briefly how to secure logs. In this post, we will dig deep into the world of logs.

Log files monitoring can be used for early detection of intrusion if it is handled perfectly, so let’s handle it.

 

The Logging Service

Most Linux distros come with Rsyslog (successor version of syslog) preinstalled as well as the logging component of systemd which is systemdjournald (journald).

Regardless of the software, they are the same; the difference is in some features.

Rsyslog Daemon

The rsyslog utility is used to create and store readable event notification messages so system administrators can manage their systems.

The Rsyslog daemon is a passive tool, it hardly waits for input from devices or programs. It actively gathers messages.

you can check if the service is running or not:

$ systemctl status rsyslog

Rsyslog can send its output to various destinations like:

  • Text files as /var/log/* files
  • SQL databases
  • Different hosts

Supported databases are MySQL, PostgreSQL, Oracle, SQLite, Microsoft SQL, Sybase, Firebird, and mSQL.

Each log entry contains the date, time, hostname, process name, PID, and the message from that process.

Configuring Rsyslog

The Rsyslog daemon configuration file is  /etc/rsyslog.conf.

Some Linux distros like Ubuntu or Linux Mint, you will find these lines on /etc/rsyslog.d/default.conf

Each line contains a selector field and an action field.

mail.*        /var/log/mail

The facility here is mail and the priority is the asterisk (*), so all messages from cron services will be logged to /var/log/mail.

We need to know what kind of facilities we have on our system and what are the other priorities.

Rsyslog Port

Rsyslog uses TCP on port 514. And its TCP because of the @@ sign.

You can ensure your rsyslog port using the netstat command:

$ netstat -tnlp | grep rsyslog

Rsyslog Facilities

The source of the Rsyslog message is called the facility. There are some Linux functions, daemons and other applications have facilities attached to them.

The following list is the rsyslog facilities in Linux:

auth                                       Security related messages.

auth-priv                             Access control messages.

cron                                       Message generated by cron subsystem.

daemon                               System daemons and process messages.

kern                                       Kernel messages.

mail                                       Mail messages.

syslog                                    Syslog messages.

user                                       The default facility when no facility is specified.

You have two additional special facilities the asterisk (*) which means all facilities and (none) which means no facility at all.

Look at the following examples to understand these special facilities.

*.emerg    /var/log/emerg

This line says send all messages of the emergency priority to /var/log/emerg file.

kern.none /var/log/messages

This will tell rsyslog not to log any kernel messages to the file /var/log/messages.

Rsyslog Levels (Priorities)

Linux syslog server Priorities are the scale of importance. We have notice, debug, info, warning, err, crit, alert, and emerg.

As with facilities, you can use the asterisk for all priorities (*) and none for no one.

Also, you can use the equal sign (=) and exclamation mark(!).

For example, kern.=crit indicates that only kernel facility messages of crit priority are to be selected.

The ! Modifier has the opposite effect; for example, kern.!crit selects all kernel facility messages except those of crit priority.

Rsyslog actions

Every event notification received by the Linux syslog server goes to the specified action, we saw the logs go to files in the previous examples, but there are more actions can be done.

Rsyslog can do the following actions:

  • Logging to a file.
  • Logging to a device file.
  • Sending to the user screen.
  • Send to named pipes.
  • Send to the remote host.

Look at the following examples to understand the difference between them:

auth.err                /var/log/messages

daemon.*           /dev/lpr2

auth-priv              root,likegeeks

kern.crit               |/var/log/mypipe

mail                       @@likegeeks.local

auth.*                   >dbhost,dbname,dbuser,dbpassword;dbtemplate

In the 1st line sends all auth messages of err priority logged to /var/log/messages file.

The 2nd line sends all daemon messages of all priorities are sent to a local printer lpr2.

The 3rd line sends all auth-priv messages to both users root and likegeeks if they are logged in.

The 4th line sends all kernel messages of crit priority to the named pipe /var/log/mypipe, you can create your named pipe using the mkfifo command.

The 5th line sends all mail messages to the host likegeeks.local on TCP port 514. This requires Rsyslog daemon to start with -r option; otherwise, the port will not open.

You should use double @ sign @@likegeeks.local, that means logs will be transmitted using TCP protocol.

The sixth line sends all auth messages to a database table with the specified parameters, and the dbtemplate field is optional.

Keep in mind that opening the port 514 on your system is dangerous, since the Rsyslog daemon is not selective about where it receives messages from, and maybe someone can flood your system with messages.

If you want remote logging it is recommended to use syslog-NG which we will discuss later on this post.

You may notice a line like the following in /etc/rsyslog.conf

mail.*    -/var/log/maillog

This tells Rsyslog not to sync the file after writing to it. This is used to speed up the process of writing to the log, but if your system crashes between write processes, you will lose data.

Combining Rsyslog Selectors

The facility and priority together make up the selector.

You can combine selectors in your rsyslog.conf file like this:

mail,kern.emerg           /var/log/log

I think it’s easy to understand this line.

All mail and all kern messages with an emerg or higher priority will be sent to /var/log/log file.

Rsyslog Filters

You can use selectors for filtering, but what if you want to filter the messages?

You can filter your messages with property-based filters like this:
:property, compare-operation, "value"
The following examples show how to filter messages that contain error
:msg, contains, "error"
The compare operations are:
contains, isequal, startswith, regex, ereregex.

:msg, regex, "fatal .* error"

This filter says catch any message that contains the words “fatal” and “error” with anything in between.

Systemd-journald Service

Systemd-Journald is a component of systemd that runs as a service to collect log data just like Rsyslog.

Systemd-journald creates its logging data differently from the traditional method of using normal text files. It stores logging data in indexed journals, which makes it faster than other tools.

Many Linux distros come with systemd-journald integrated alongside with Rsyslog.

You can run them concurrently on the same system and even feed the data from one to the other.

The main tool for interacting with journaled files is journalctl.

Journalctl is used for querying and viewing the contents of logging data collected by systemd-journald service.

$ journalctl

Linux Syslog Server journalctl

You can use your keyboard to navigate through messages and press q to quit.

Some of the output lines are colored and others are bold. The red color for the priority of crit [2], alert [1], and emerg [0].

The red color and bold for the priority of warn [4], crit [2], alert [1], and emerg [0].

The lines of priority debug (7) and higher info [6] are displayed normally.

The best thing about journalctl command is that you can filter the messages with some flexible options.

You can display a specific number of lines like this:

$ journalctl -n 3

Linux Syslog Server journalctl -n

You can show real-time messages generated like this:

$ journalctl -f

Also, you can show messages since a number of days:

$ journalctl --since="3 days ago"

Or you can show messages between days like this:

$ journalctl --since="5 days ago" --until="2 days ago"

Also, you can specify the date you want:

$ journalctl --since="2017-03-14"

Maybe you want at a specific time:

$ journalctl --since="2017-03-14 15:00" --until="2017-03-14 15:30"

And you can specify the priority you want:

$ journalctl --since="2017-03-14 15:00" --until="2017-03-14 15:30" -p warning

You can display messages associated with a specific user like this:

$  journalctl _UID=1001

You can view messages generated by a program like /sbin/init:

$ journalctl /sbin/init

Also, you can check the messages by your disk:

$ journalctl /dev/sda1

Syslog-ng

Another successor to syslog which is syslog-ng. This tool has great flexibility and considerably more regard for security.

Moreover, syslog-ng allows for more advanced message filtering, manipulation, and interaction.

Some people prefer syslog-ng because it has cleaner syntax than Rsyslog.

The configuration file for syslog-ng is /etc/syslog-ng/syslog-ng.conf  file.

This file contains the following blocks:

options{}                              Global options.

source{}                               Statements defining where messages are coming from.

destination{}                      Statements defining where messages are sent or stored.

filter{}                                   Filtering statements.

log{}                                       Statements that do the actual logging.

The idea in syslog-ng is to write the source{},destination{} and filter{} then you use them in the log{} statement. We will see how to write each one of them and how to write the final log statement.

Syslog-ng Source Statements

You can write source statements like this:

source my_tcp { tcp(ip(192.168.1.5) ;};

The previous tcp() source statements specify 192.168.1.5 as the IP address to which syslog-ng should bind.

You can also specify a file as a source of logging:

source my_file { file("/proc/kmsg"); };

Also, you can specify named pipes as a source:

source my_pipe { pipe("/var/mypipe"); };

Syslog-ng Destination Statements

The destination statement includes all the lines that tell syslog-ng where to put its output.

You can write destination statements like that:

destination my_dest { file("/var/log/messages" owner(root) group(root) perm(0644)); };

You specify the name of the file and other options like the ownership of the file itself.

The destination can be named pipe like this:

destination my_dest { pipe("/var/mypipe"); };

You can use the ready to use file expansion macros like this:

destination my_dest { file("/var/log/hosts/$HOST/$FACILITY$YEAR$MONTH$DAY"); };

The destination can be a logged in user like this:

destination my_dest { usertty("root"); };

Syslog-ng Filters Statements

This is similar to the facility and priority selector used by the Rsyslog daemon as we saw before.

filter my_filter { facility(kern); priority(notice .. crit) };

We’ve select all messages from the facility kern using the facility() option and with a range of priorities from notice to crit using the priority() option.

The priority range is separated by two dots (..)

You can filter by host:

filter my_filter { host(likegeeks); };

Or by specific program:

filter my_filter { program("telnet.*") };

Or even use regular expressions to filter messages:

filter my_filter { match("YOUR REGEX"); };

And even negate the regular expressions:

filter my_filter { not match("YOUR REGEX"); };

Syslog-ng Log Statements

Now we have the source, destination, and filter statements, we should write the log statements that will do the actual logging.

The log statements don’t need to be named, unlike other statements, since they will not be used elsewhere.

log { source(my_src); destination(my_dest); };

This line sends all the messages from the source my_src to the destination my_dest.

You are not limited to one source, you can combine multiple sources.

log { source(my_src); source(my_kern); filter(my_filter); destination(my_dest); };

Awesome!!

Logging syslog-ng Messages in SQL Database

Maybe you like to query your logs using powerful SQL statements, so you need to send the logs to a database.

Syslog-ng supports MSSQL, MySQL, Oracle, PostgreSQL, and SQLite databases.

This example sends the log messages to a MySQL database running on the localhost. The messages are inserted into the logs database.

The syslog-ng automatically creates the necessary tables and columns.

Then you can use this destination on any log statement you want.

Log Files Locations

The default log files locations that you might need to take a look at:

/var/log/messages                           Contains general system messages.

/var/log/kern.log                              Kernel logs.

/var/log/cron                                     The cron daemon messages.

/var/log/btmp                                   Contains failed login attempts.

/var/log/mail.log                               Contains logs for the mail server.

/var/log/wtmp                                   Contains all login and logout history.

/var/run/utmp                                   Contains the login state of each user.

/var/log/dmesg                                 This contains very important messages about the kernel

/var/log/secure                                 Security related messages will be logged here.

/var/log/mariadb                              MariaDB log directory.

/var/log/mysql                                   MySQL log directory.

/var/log/httpd/                                  Apache web-server log directory.

for CPanel apache logs are at this location

/usr/local/apache/logs/                   General Apache logs.

/usr/local/apache/domlogs/           Domain specific logs.

/var/log/exim/                                   Exim mail server logs.

/var/log/yum.log                               Logs from the Yum package manager.

/var/log/boot.log                              Contains information about when the system boots.

Using Logrotate

It’s very important to keep your log file at a manageable size so you can control it.

If you need to store log messages for the long term, it is preferred to use a database as we saw.

Log rotation can be quite complicated if you do it manually, it’s recommended to use the logrotate tool.

logrotate configuration is in  /etc/logrotate.conf file.

If you look at this file, you’ll see that all log files rotate weekly, logs are rotated four times before they are deleted, new log files are created after rotating old ones.

You can modify the log rotation process the way you want. Also, you can include your log files.

The following options can be used in your logrotate.conf file:

daily                                     Logs are rotated on a daily basis.

weekly                                 Logs are rotated on a weekly basis.

monthly                              Logs are rotated on a monthly basis.

compress                            Old log files are compressed with gzip.

include                                To include more conf file.

You can check the shell script that runs daily for log rotation in /etc/cron.daily/logrotate.

I hope you find the post useful. The Linux syslog server and the tools used to manage logs make the life easy for system administrators.

Thank you.