Linux Syslog Server
Server Administration

Linux Syslog Server and Log Management

In this post, we will talk about Linux Syslog Server and how to manage your logs.

If you want your system secured and hardened, you have to know what’s going on in that system, you can do that using logs. With logs, you can diagnose problems and determine the status of your system and applications.

When it comes to security you have to dig deeper into logging to understand every aspect of your system and what threats might affect your system.

In a previous post, we talked about how to secure Linux server and we mentioned briefly how to secure logs, in this post, we will dig deep into the world of logs.

Logs are very important for two main reasons, the first reason, logs can be an entry point for the attacker to penetrate your system, and some attackers target your logs to find any useful information about your system.

The second reason, if the attacker gains access to your system, the last thing he wants to do is to clean everything after him and this can be done by deleting the logs or whitewash his steps from log files, so you have no evidence about what happened.

Log files monitoring can be used for early detection of intrusion tool if it is handled perfectly.

In this post, we are going to cover Linux Syslog server, since Syslog format being the closest thing to a universal logging standard that exists.

 

The Logging Service

Most Linux distros come with rsyslog (successor version of syslog) preinstalled as well as the logging component of systemd which is systemdjournald (journald).

Regardless of the software, they are the same; the difference is in some features.

Rsyslog daemon maintains backward compatibility with the traditional syslog daemon with some additional features.

Rsyslog Daemon

The rsyslog utility is used to generate, process, and store meaningful event notification messages that provide the information required for administrators to manage their systems.

The rsyslog daemon is a passive tool. It merely waits for input from devices or programs. It actively gathers messages.

you can check if the service is running or not:

$ systemctl status rsyslog

rsyslog can send its output to various destinations like:

  • Text files as /var/log/* files
  • SQL databases
  • Different hosts

Supported databases are MySQL, PostgreSQL, Oracle, SQLite, Microsoft SQL, Sybase, Firebird, mSQL.

Each log entry consists of a single line containing the date, time, hostname, process name, PID, and the message from that process.

Configuring Rsyslog

The rsyslog daemon is controlled by a configuration file located in /etc/rsyslog.conf.

This file contains the information about what devices and programs rsyslog is listening for

Some Linux distros like Ubuntu or Linux mint you will find these lines on /etc/rsyslog.d/default.conf

Each line is structured into two fields, a selector field, and an action field.

cron.*   /var/log/cron

This example shows a facility and priority selector, cron.*, together with the action /var/log/cron. The facility represented here is cron and the priority is the asterisk (*), so all messages from cron services will be logged to /var/log/cron.

So there is a facility and priority from that facility goes to the action file.

We need to know what kind of facilities we have on our system and what the other priorities are.

Rsyslog Port

rsyslog defaults to using TCP on port 514. In the configuration file, /etc/rsyslog.conf, TCP is indicated by @@.

You can ensure your rsyslog port using the netstat command:

$ netstat -tnlp | grep rsyslog

Rsyslog Facilities

The source of the rsyslog message is called the facility. There are some Linux functions, daemons and other applications have facilities attached to them.

The following list is the rsyslog facilities in Linux:

auth                                       Security related messages

auth-priv                             Access control messages

cron                                       message generated by cron subsystem

daemon                               System daemons and process messages

kern                                       Kernel messages

local0–local9                      Reserved for locally defined messages

lpr                                          printer subsystem messages

mail                                       Mail-related messages

news                                     Network News–related messages

syslog                                    Syslog-related messages

user                                       The default facility when no facility is specified

You have two additional special facilities the asterisk (*) which means all facilities and (none) which means no facility at all.

Look at the following examples to understand those special facilities.

*.emerg    /var/log/emerg

This will send all messages of the emerg priority, regardless of the facility to /var/log/emerg file.

kern.none /var/log/messages

This will tell rsyslog to not log any kernel messages to the file /var/log/messages.

Rsyslog Levels (Priorities)

Linux syslog server Priorities are the scale of importance. We have debug, info, notice, warning, err, crit, alert, and emerg.

As with facilities, you can use the asterisk for all priorities (*) and none.

You can use two other modifiers (=) and (!).

The = modifier indicates that only one priority is selected.

For example, kern.=crit indicates that only kernel facility messages of crit priority are to be selected.

The ! Modifier has the opposite effect; for example, kern.!crit selects all kernel facility messages except those of crit.

Rsyslog actions

Every event notification received by the Linux syslog server goes to the specified action, we’ve seen the logs goes to files in the previous examples, but there are more actions can be done.

Rsyslog can do the following actions:

  • logging to a file
  • logging to a device file
  • sending to the user screen
  • send to named pipes
  • send to the remote host

Look at the following examples to understand the difference between them:

auth.err                /var/log/messages

daemon.*           /dev/lpr2

auth-priv              root,likegeeks

kern.crit               |/var/log/mypipe

mail                       @@likegeeks.local

auth.*                   >dbhost,dbname,dbuser,dbpassword;dbtemplate

In the first line sends all auth messages of err priority logged to /var/log/messages file.

The second line sends all daemon messages of all priorities are sent to a local printer lpr2.

The third line sends all auth-priv messages to the users root and likegeeks if they are logged in.

The fourth line sends all kernel messages of crit priority to the named pipe /var/log/mypipe, you can create your own named pipe using mkfifo command.

The fifth line sends all mail messages to the host likegeeks.local on TCP port 514. This required rsyslog daemon to start with -r option; otherwise the port will not open.

You should use double @ sign @@likegeeks.local, that means logs will be transmitted using TCP protocol.

The sixth line sends all auth messages to a database table with the specified parameters, and the dbtemplate field is optional.

Keep in mind that opening the port 514 on your system is dangerous, since the rsyslog daemon is not selective about where it receives messages from, and maybe someone could flood your system with messages.

If you want remote logging it is recommended to use syslog-NG which we will discuss later.

You may notice a line like the following in /etc/rsyslog.conf

mail.*    -/var/log/maillog

This tells rsyslog not to sync the file after writing to it. This is used to speed up the process of writing to the log. But if your system crashes between write attempts, you will lose data.

Combining Rsyslog Selectors

The facility and priority together makes up the selector.

You can combine selectors in your rsyslog.conf file like this:

auth,auth-priv.crit           /var/log/auth

This line sends all auth messages and all auth-priv messages with a priority of crit or higher to /var/log/auth file.

Rsyslog Filters

You can use selectors for filtering, but what if you want to filter the messages?

You can filter your messages with property-based filters like this:
:property, compare-operation, "value"
The following examples show how to filter messages that contain error
:msg, contains, "error"
The compare operations are:
contains, isequal, startswith, regex, ereregex
The difference between regex and ereregex is:
regex compares POSIX BRE regular expression
ereregex compares POSIX ERE regular expression

:msg, regex, "fatal .* error"
This filter matches when the string contains the words “fatal” and “error” with anything in between.

Systemd-journald Service

Systemd-Journald is a component of systemd that runs as a service to collect log data just like rsyslog.

Systemd-journald creates its logging data differently from the traditional method of using normal text files. It stores logging data in indexed journals, which makes it faster than other tools.

Many Linux distros come with systemd-journald integrated alongside with rsyslog.

You can run them concurrently on the same system and even feed the data from one to the other.

The main tool for interacting with journaled files is journalctl.

Journalctl is used for querying and viewing the contents of logging data collected by systemd-journald service.

$ journalctl

Linux Syslog Server journalctl

You can use your keyboard to navigate through messages and press q to quit.

Some of the output lines are colored and other are bold. The red color for the priority of crit [2], alert [1], and emerg [0].

The red color and bold for the priority of warn [4], crit [2], alert [1], and emerg [0]

The lines of priority debug (7) and higher info [6] are displayed normally.

The best thing about journalctl command is that you can filter the messages with some flexible options.

You can display a specific number of lines like this:

$ journalctl -n 3

Linux Syslog Server journalctl -n

You can show real-time messages generated like this:

$ journalctl -f

Also, you can show messages since a number of days:

$ journalctl --since="3 days ago"

Or you can show messages between 5 days and 2 days:

$ journalctl --since="5 days ago" --until="2 days ago"

Also, you can specify the date you want:

$ journalctl --since="2017-03-14"

Maybe you want at a specific time:

$ journalctl --since="2017-03-14 15:00" --until="2017-03-14 15:30"

And you can specify the priority you want:

$ journalctl --since="2017-03-14 15:00" --until="2017-03-14 15:30" -p warning

You can display messages associated with a specific user like this:

$  journalctl _UID=1001

You can view messages generated by a program like /sbin/init:

$ journalctl /sbin/init

Also, you can check the messages by you disk:

$ journalctl /dev/sda1

Syslog-ng

Another successor to syslog which is syslog-ng. This tool has great flexibility and considerably more regard for security.

Additionally, syslog-ng allows for more sophisticated message filtering, manipulation, and interaction.

Some people prefer syslog-ng because it has cleaner syntax than rsyslog.

syslog-ng goes through a lot of active development, and new features are added all the time, you can check the git repo, you’ll see that the last commit is two days ago since this post date.

The configuration file for syslog-ng is /etc/syslog-ng/syslog-ng.conf  file.

This file contains the following blocks:

options{}                              Global options.

source{}                               Statements defining where messages are coming from.

destination{}                      Statements defining where messages are sent or stored.

filter{}                                   Filtering statements.

log{}                                       Statements that do the actual logging.

The idea in syslog-ng is to write the source{},destination{} and filter{} then you use them in the log{} statement, and we will see how to write each one of them and how to write the final log statement.

Syslog-ng Source Statements

You can write source statements like this:

source my_tcp { tcp(ip(192.168.1.5) ;};

The previous tcp() source statements specify 192.168.1.5 as the IP address to which syslog-NG should bind.

You can also specify a file as a source of logging:

source my_file { file("/proc/kmsg"); };

Also, you can specify named pipes as a source:

source my_pipe { pipe("/var/mypipe"); };

Syslog-ng Destination Statements

The destination line includes all the lines that tell syslog-ng where to put its output.

You can write destination statements like that:

destination my_dest { file("/var/log/messages" owner(root) group(root) perm(0644)); };

You specify the name of the file and other options like the ownership of the file itself.

The destination can be named pipe like this:

destination my_dest { pipe("/var/mypipe"); };

You can use the ready to use file expansion macros like this:

destination my_dest { file("/var/log/hosts/$HOST/$FACILITY$YEAR$MONTH$DAY"); };

The destination could be a logged in user like this:

destination my_dest { usertty("root"); };

Syslog-ng Filters Statements

This is much like the facility and priority selector used by the rsyslog daemon as we’ve in before.

filter my_filter { facility(kern); priority(info .. emerg) };

We’ve select all messages from the facility kern using the facility() option and with a range of priorities from info to emerg using the priority() option.

The priority range is separated by two dots (..)

You can filter by host:

filter my_filter { host(likegeeks); };

Or by specific program:

filter my_filter { program("sshd.*") };

Or even use regular expressions to filter messages:

filter my_filter { match("YOUR REGEX"); };

And even negate the regular expressions:

filter my_filter { not match("YOUR REGEX"); };

Syslog-ng Log Statements

Now we have the source, destination and filter statements, we should write the log statements that will do the actual logging.

The log statements don’t need to be named, unlike other statements, since they will not be used elsewhere.

log { source(my_src); destination(my_dest); };

This line sends all the messages from the source my_src to the destination my_dest.

You are not limited to one source, you can combine multiple sources.

log { source(my_src); source(my_kern); filter(my_filter); destination(my_dest); };

Awesome!!

Logging syslog-ng Messages in SQL Database

Maybe you like to query your logs using powerful SQL statements, so you need to send the logs to a database.

Syslog-ng supports MSSQL, MySQL, Oracle, PostgreSQL, and SQLite databases.

This example sends the log messages to a MySQL database running on the localhost. The messages are inserted into the logs database. The syslog-ng application automatically creates the required tables and columns, if the user account used to connect to the database has the required privileges.

Then you can use this destination on any log statement you want.

Log Files Locations

The default log file locations that you might need to take a look at:

/var/log/messages                           Contains general system messages.

/var/log/kern.log                              Kernel logs

/var/log/cron                                     The cron daemon messages.

/var/log/mail.log                               Log information from the mail server.

/var/log/wtmp                                   Contains all login and logout history.

/var/log/btmp                                   Records failed login attempts.

/var/run/utmp                                   Logs the present login state of each user.

/var/log/dmesg                                 This contains very important messages about the kernel

/var/log/secure                                 Security related messages will be logged here.

/var/log/mariadb                              MariaDB log directory

/var/log/mysql                                   MySQL log directory.

/var/log/httpd/                                  Apache web-server log directory.

for CPanel apache logs are at this location

/usr/local/apache/logs/                   General Apache logs.

/usr/local/apache/domlogs/           Domain specific logs.

/var/log/exim/                                   Exim mail server logs.

/var/log/yum.log                               Logs from the Yum package manager.

/var/log/boot.log                              Contains information about when the system boots

Using Logrotate

It’s very important to keep your log file at a manageable size so you can control it.

If you need to store log messages for the long term, it is preferred to use a database as we’ve seen.

Log rotation can be quite complicated if you do it manually, it’s recommended to use the logrotate tool.

logrotate configuration is in  /etc/logrotate.conf file.

If you look at this file you’ll see that all log files rotate weekly, logs are rotated four times before they are deleted new log files are created after rotating old ones.

You can modify the log rotation process the way you want. Also, you can include your log files.

The following options can be used in your logrotate.conf file:

daily                                     Logs are rotated on a daily basis.

weekly                                 Logs are rotated on a weekly basis.

monthly                              Logs are rotated on a monthly basis.

compress                            Old log files are compressed with gzip.

include                                To include more conf file.

You can check the shell script that runs daily for log rotation in /etc/cron.daily/logrotate.

I hope you find the post useful. The Linux syslog server and the tools used to manage logs make the life easy for system administrators.

Thank you.