Linux Security - Forwarding the Journal logs

Recently I wrote an article about how to analyse the Systemd Journal during incident response. There was a follow-up discussion about how to make sure this data was moved to a log centralisation platform/SIEM. Now, forwarding the data is definitely a good idea as it allows you to analyse it remotely and reduces the risk that an attacker can tamper with it.

This article is a quick summary of how you can configure a Linux system to forward the logs, using either Rsyslog or Syslog-ng. I am going to use Ubuntu as the example system but it is pretty similar when using RHEL/SLES etc.

Note: This focuses on the journal, but you can also use it to forward any data you've sent to Syslog.

Forwarding the journal with rsyslog

Step 1. Make sure the journal forwards logs to rsyslog

Edit the journald.conf file to uncomment the entry which reads "ForwardToSyslog=yes" and then restart the journal. For example:

sudo nano /etc/systemd/journald.conf        

sudo systemctl restart systemd-journald        

You might want to also use this time to look at other ways to optimise your journal, but if you are confident this is the only change, you can even roll this into a single-line command:

sudo sed -i 's/^#\(ForwardToSyslog=yes\)/\1/' /etc/systemd/journald.conf && sudo systemctl restart systemd-journald        

You now have a journal that is sending data to syslog! What we need to do next is make syslog (or rsyslog in this case) send that data to where we need it.

Step 2. Configure rsyslog to forward the event data

This involves modifying Ryslog.

Open the rsyslog configuration file, ensure that the imuxsoc and imjournal modules are loaded (uncommented) and then add a line at the end which forwards data to the collector.

First, edit the config file:

sudo nano /etc/rsyslog.conf        

Ensure that the following lines are present - and not commented out. If they are missing, then please add them.

Next, add this string to the end of the ryslog.conf file (replace 10.10.10.10 with the IP address of your centralised log storage device)

Finally, restart rsyslog

sudo systemctl status rsyslog        

Step 3. Check things are working.

On the source system, you can verify this as follows

Check that rsyslog is working:

sudo systemctl status rsyslog        

It should look something like this:

You can also check that traffic is flowing - although this will need you to generate event data. A quick and simple approach is to use tcpdump. (Make sure the IP address relates to your system)

sudo tcpdump -i any host 10.10.10.10 and port 514        

Finally, you can use the logger command to send a test message and check it is received on the correlation system.

logger "your message here"        

And then check it has been received on the destination system. How you do this depends on the destination system but examples include:

tail -f /var/log/syslog
tail -f /var/log/messages        

You have now successfully configured the Linux journal to be logged to a remote log collector.

Forwarding the journal with Syslog-ng

Syslog-ng users can have effectively the same outcome, but there are some minor changes. The journal configuration remains the same but this time, modify the syslog-ng.conf file and add the remote destination data.

sudo nano /etc/syslog-ng/syslog-ng.conf        

sudo systemctl restart syslog-ng        

Then test the connection as mentioned previously.

What about other distros?

The examples here are from Ubuntu, so you might be curious about how to get it working on RHEL, SUSE etc.

The good news is that the syntax is pretty much identical. The only noticeable difference would be if you needed to install Rsyslog or Syslog-ng then you'd use dnf or yum (etc). The configuration options remain consistent across platforms.

Summary

The journal is very much the future of event data on Linux systems. By default, this is only written locally to a binary file, and this is less than ideal for incident response purposes. With a few simple configuration steps, you can send this data to a remote collector, so even if you dont have a fully configured SIEM, you can have a central analytics point.

And yes, it is better if you have a SIEM in place.