Linux Syslog Server and Log Management

In this post, we will talk about Linux Syslog Server and how to manage your logs.

If you want to secure your system, you have to know what’s going on in that system; you can do that using logs. With logs, you can diagnose problems and determine the health of your system and applications.

In this post, we will dig deep into the world of logs.

You can monitor log files for early detection of intrusion if it happened, so let’s handle it.

 

 

The logging service

Most Linux distros come with RSyslog (successor version of syslog) preinstalled as well as the logging component of systemd which is systemd–journald (journald).

Regardless of the software, they are the same; the difference is in some features.

 

RSyslog daemon

You can use the RSyslog utility to create and store readable event notification messages so system administrators can manage their systems.

you can check if the service is running or not:

$ systemctl status rsyslog

RSyslog can send its output to various destinations like:

• Text files as /var/log/* files
• SQL databases
• Different hosts

Supported databases are MySQL, PostgreSQL, Oracle, SQLite, Microsoft SQL, Sybase, Firebird, and mSQL.

Each log entry contains the date, time, hostname, process name, PID, and the log message.

 

Configuring RSyslog

The RSyslog daemon configuration file is /etc/rsyslog.conf.

Some Linux distros like Ubuntu or Linux Mint, you will find these lines in

/etc/rsyslog.d/default.conf

Each line contains a selector and an action.

kern.*        /var/log/messages

The kernel is the facility, and the priority is the asterisk (*), so the kernel will send all messages to /var/log/messages.

We need to know what kind of facilities we have on our system and what are the other priorities.

 

RSyslog port

RSyslog uses TCP on port 514. And its TCP because of the @@ sign.

You can ensure your RSyslog port using the netstat command:

$ netstat -tnlp | grep rsyslog

 

RSyslog facilities

We call the source of the RSyslog message a facility. There are some Linux functions, daemons, and other applications that have facilities attached to them.

The following list is the RSyslog facilities in Linux:

auth	Security-related messages.
auth-priv	Private authentication messages.
cron	Message generated by the cron subsystem.
daemon	Services messages.
kern	Kernel messages.
mail	Mail messages.
syslog	Syslog messages.

You have two additional special facilities, the asterisk (*), which means all facilities, and (none), which means no facility at all.

Look at the following examples to understand these special facilities.

*.emerg    /var/log/emerg

This line says to send all messages of the emergency priority to /var/log/emerg file.

mail.none /var/log/maillog

This will tell RSyslog not to log any mail messages to the file /var/log/maillog.

 

RSyslog levels (Priorities)

Linux syslog server Priorities are the scale of importance. We have notice, debug, info, warning, err, crit, alert, and emerg.

You can use the asterisk for all priorities (*) and none for no one.

Also, you can use the equal sign (=) and exclamation mark(!).

For example,

kern.=crit

Here we select the critical kernel messages.

kern.!crit

And here, we select all kernel messages except the critical priority.

 

RSyslog actions

Every event notification received by the Linux syslog server goes to the specified action, we saw the logs go to files in the previous examples, but we can do more actions.

RSyslog can do the following actions:

• Logging to a file.
• Logging to a device file.
• Sending to the user screen.
• Send to named pipes.
• Send to the remote host.

Look at the following examples to understand the difference between them:

auth.err	/var/log/messages
daemon.*	/dev/lpr2
auth-priv	root,likegeeks
kern.crit	|/var/log/mypipe
mail	@@likegeeks.local
auth.*	>dbhost,dbname,dbuser,dbpassword;dbtemplate

In the 1st line sends all auth messages of err priority logged to /var/log/messages file.

The 2nd line sends all daemon messages of all priorities to a local printer lpr2.

The 3rd line sends private authentication messages to both users root and likegeeks if they are logged in.

The 4th line sends all kernel messages of crit priority to the named pipe /var/log/mypipe. You can create your named pipe using the mkfifo command.

The 5th line sends all mail messages to the host likegeeks.local on TCP port 514. This requires RSyslog daemon to start with -r option; otherwise, the port will not open.

You should use double @ sign @@likegeeks.local, which means RSyslog will use TCP protocol to transmit logs.

The sixth line sends all auth messages to a database table with the specified parameters, and the dbtemplate field is optional.

Keep in mind that opening the port 514 on your system is dangerous, the attacker can flood your system with messages.

If you want remote logging, you should use syslog-NG, which we will discuss later on this post.

You may notice a line like the following in

/etc/rsyslog.conf

mail.*    -/var/log/maillog

This tells RSyslog to cancel file synchronization after writing logs, but you may lose data if the writing process doesn’t complete successfully.

 

Combining RSyslog selectors

The facility and priority together make up the selector.

You can combine selectors in your rsyslog.conf file like this:

mail,kern.emerg           /var/log/log

I think it’s easy to understand this line.

Here we will send all mail and all kern messages with an emerg or higher priority to /var/log/log file.

 

RSyslog filters

You can use selectors for filtering, but what if you want to filter the messages?

You can filter your messages with property-based filters like this:

:property, compare-operation, "value"

The following examples show how to filter messages that contain an error:

:msg, contains, "error"

The compare operations are:
contains, isequal, startswith, regex, ereregex.

:msg, regex, "login .* failed"

This filter says catch any message that contains the words “login” and “failed” and any text between them.

 

Systemd-journald service

Systemd-Journald is a component of systemd that runs as a service to collect log data just like RSyslog.

It stores logging data in indexed journals, which makes it faster than other tools.

Many Linux distros come with systemd-journald integrated alongside with RSyslog.

You can run them concurrently on the same system and even feed the data from one to the other.

The main tool for interacting with journaled files is journalctl.

You can use the journalctl to query and view the contents of logging data that systemd-journald service collects.

$ journalctl

If you want to navigate through messages, you can use your keyboard, and after you finish, press q to quit.

You may notice some lines with colors, and others are bold. The red color for the priority of crit [2], alert [1], and emerg [0].

The red color and bold for the priority of warn [4], crit [2], alert [1], and emerg [0].

The lines of priority debug (7) and higher info [6] are displayed normally.

The best thing about journalctl command is that you can filter the messages with some flexible options.

You can display a specific number of lines like this:

$ journalctl -n 3

You can show real-time messages generated like this:

$ journalctl -f

Also, you can show messages since a number of days:

$ journalctl --since="3 days ago"

Or you can show messages between days like this:

$ journalctl --since="5 days ago" --until="2 days ago"

Also, you can specify the date you want:

$ journalctl --since="2017-03-14"

Maybe you want at a specific time:

$ journalctl --since="2017-03-14 15:00" --until="2017-03-14 15:30"

And you can specify the priority you want:

$ journalctl --since="2017-03-14 15:00" --until="2017-03-14 15:30" -p warning

You can display messages associated with a specific user like this:

$  journalctl _UID=1001

You can view messages generated by a program like /sbin/init:

$ journalctl /sbin/init

Also, you can check the messages on your disk:

$ journalctl /dev/sda1

 

Syslog-ng

Another successor to syslog which is syslog-ng. This tool is better and more secure.

Moreover, syslog-ng allows for more advanced message filtering, manipulation, and interaction.

Some people prefer syslog-ng because it has a cleaner syntax than RSyslog.

The configuration file is

/etc/syslog-ng/syslog-ng.conf

file.

This file contains the following blocks:

options{}	Global options.
source{}	Define the source of the messages.
destination{}	Define the destination of the messages.
filter{}	Defines a filter to filter the messages.
log{}	Write the logs from the source to the destination specified from the above statements.

The idea in syslog-ng is to write the source{},destination{} and filter{} then you use them in the log{} statement. We will see how to write each one of them and how to write the final log statement.

Syslog-ng source statements

You can write source statements like this:

source my_udp { udp(ip(1.2.3.4) ;};

This line specifies the source of messages, which is the IP address 1.2.3.4.

You can also specify a file as a source of logging:

source my_file { file("/proc/kmsg"); };

Also, you can specify named pipes as a source:

source my_pipe { pipe("/var/mypipe"); };

Syslog-ng destination statements

You can write destination statements like that:

destination my_dest { file("/var/log/mylogs" owner(likegeeks) group(likegeeks) perm(0644)); };

You specify the filename and its ownership.

Here we will use a named pipe as a source:

destination my_dest { pipe("/var/mypipe"); };

You can use the ready to use file expansion macros like this:

destination my_dest { file("/var/log/hosts/$HOST/$FACILITY$YEAR$MONTH$DAY"); };

The destination can be a logged-in user like this:

destination my_dest { usertty("root"); };

Syslog-ng filters statements

You can define a filter by specifying the facilities and the priorities that you want to select like this:

filter my_filter { facility(kern); priority(notice .. crit) };

The above filter statement selects all messages from the kernel with priorities from notice to crit.

The priority range is separated by two dots (..)

You can filter by host:

filter my_filter { host(likegeeks); };

Or by specific program:

filter my_filter { program("telnet.*") };

Or even use regular expressions to filter messages:

filter my_filter { match("YOUR REGEX"); };

And even negate the regular expressions:

filter my_filter { not match("YOUR REGEX"); };

Syslog-ng log statements

Now we have the source, destination, and filter statements. We should write the log statements that will do the actual logging.

You don’t need to name the log statements, unlike other statements, since they will not be used elsewhere.

log { source(my_src); destination(my_dest); };

This line sends the messages from my_src to my_dest.

The above statement uses one source, what about combining multiple sources?

log { source(my_src); source(my_kern); filter(my_filter); destination(my_dest); };

Awesome!!

 

Logging syslog-ng messages in SQL database

Syslog-ng supports many databases as a backend. We will use MySQL in the following example.

destination my_dest {
sql(type(mysql)
host("localhost") username("likegeeks") password("likegeeks")
database("dblogs")
table("logs")
columns("datetime", "host", "program", "pid", "logmessage")
values("$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC", "${HOST}", "${PROGRAM}", "${PID}", " $MSG")
indexes("datetime", "host", "program", "pid", "logmessage"));
};

This example sends the log messages to a MySQL database running on the localhost.

The syslog-ng automatically creates the necessary tables and columns.

Then you can use this destination on any log statement you want.

 

Log Files locations

The default log files locations that you might need to take a look at:

/var/log/messages	Contains general system messages.
/var/log/kern.log	Kernel logs.
/var/log/cron	The cron daemon messages.
/var/log/btmp	Contains failed login attempts.
/var/log/mail.log	Contains logs for the mail server.
/var/log/wtmp	Contains all login and logout attempts.
/var/log/dmesg	Contains kernel messages.
/var/log/secure	Contains security related messages.
/var/log/mariadb	MariaDB log directory.
/var/log/mysql	MySQL log directory.
/var/log/httpd/	Apache web-server log directory.

For CPanel apache logs are at this location:

/usr/local/apache/logs/	Apache server logs.
/usr/local/apache/domlogs/	Domain specific logs.
/var/log/exim/	Exim mail server logs.
/var/log/yum.log	Yum package manager logs.
/var/log/boot.log	Contains information about when the system boots.

 

Using Logrotate

It’s very important to keep your log file at a manageable size so you can control it.

If you need to store log messages for the long term, you should use a database as we saw.

You can use the logrotate tool for log rotation, so you keep your log files in manageable sizes.

logrotate configuration is in  /etc/logrotate.conf file.

weekly
rotate 4
create

This configuration rotates logs weekly, and after four times of rotation, the log file is deleted, then create new log files.

You can modify the log rotation process the way you want. Also, you can include your log files.

Consider using any of the following options in your logrotate.conf file:

daily	rotate daily.
weekly	rotate weekly.
monthly	rotate monthly.
compress	compress old rotated logs.
include	To include more conf files.

You can check the shell script that runs daily for log rotation in /etc/cron.daily/logrotate.

I hope you find the post useful. The Linux syslog server and the tools used to manage logs make the life easy for system administrators.

Thank you.