Remote log forwarding

IAM appliance integrates with SIEM software by forwarding its log via syslog protocol. Everything in IAM appliance goes to the rsyslog daemon which then distributes logs into separate logfiles.

This howto will show you how to enhance rsyslog configuration to also send logs over the network into your SIEM solution. You will configure simple UDP-based log forwarding or SSL-secured TCP log forwarding. In reality you can use any configuration that rsyslog can handle.

Understanding rsyslog configuration

All major components of the IAM appliance use rsyslog for logging, namely:

The operating system itself.
- This logging is the same as on any other ordinary Linux machine.
All containerized services.
- Logging through docker to rsyslog. There is specific configuration in the rsyslog which separates logs into per-service logfiles.

Rsyslog configuration of IAM appliance services is stored in multiple files in the /etc/rsyslog.d/ directory:

[root@localhost rsyslog.d]# ll
total 28
-rw-r--r--. 1 root root 144 Jan 20  2021 01_enable_udp_listen.conf
-rw-r--r--. 1 root root 176 Sep  6  2021 10_cas.conf
-rw-r--r--. 1 root root 190 Sep  6  2021 10_czechidm.conf
-rw-r--r--. 1 root root 199 Sep  6  2021 10_czechidm-db.conf
-rw-r--r--. 1 root root 214 Sep  6  2021 10_directory-server.conf
-rw-r--r--. 1 root root 193 Sep  6  2021 10_web-proxy.conf

If you look, for example, into CAS logging configuration

[root@localhost rsyslog.d]# cat 10_cas.conf
# This configuration is governed by RPM package. All manual changes will be lost.
:programname, isequal, "cas" {
	action (type="omfile" file="/data/logs/cas/cas.log")
	stop
}

you can see that whenever program name equals cas, logs are directed to the logfile and then the processing of the message stops.

UDP log forwarding

UDP-based syslog is inherently insecure because it transfers logs over the network in plaintext. It should be used only within dedicated (or otherwise secured) network segments.

Another option is to use TCP-based syslog with TLS but then you have to configure message queue on the sender (on the IAM appliance). Otherwise, when the SIEM server becomes unreachable, even processes on the sending machine may come to halt. Number of messages going through the syslog can be quite high and we recommend you perform sizing analysis beforehand.

Configuring log forwarding

If you want to, for example, configure CAS logs to also go to the SIEM, you create new configuration 09_cas_fwd.conf with following contents:

[root@localhost rsyslog.d]# cat 09_cas_fwd.conf
if $programname == "cas" then {
	action (type="omfwd" protocol="udp" target="10.0.1.247" port="514")
}

Key points here are:

This configuration is placed in 09_cas_fwd.conf which goes before 10_cas.conf. If it was loaded after 10_cas.conf, it would not work because the 10_cas.conf contains a stop clause.
The configuration in 09_cas_fwd.conf does not contain a stop clause. Logs are sent over the network according to 09_cas_fwd.conf but also further processed which also directs them into the local logfile as is specified in the 10_cas.conf.
If you want to forwarding just CAS audit logs, you need to add the AND $msg contains 'Slf4jLoggingAuditTrailManager' part to filter line. This filter works only for CAS logs.

You can modify the log forwarding configuration completely to your liking but you must not edit configuration files that come with the IAM appliance.

After you configure the rsyslog, restart it for changes to take effect.

[root@localhost ~]# systemctl restart rsyslog

Testing

Rsyslog now starts to send logs over the network. If you configured standard UDP on port 514, you can easily test with running nc on the target machine. Example of forwarded logs (newlines added for clarity).

[root@siem ~]# nc -u -l 514
<30>Jul 20 15:43:12 localhost.localdomain cas[1017]: #033[m#033[32m2022-07-20 15:43:12,399 INFO [org.apereo.cas.ticket.registry.DefaultTicketRegistryCleaner] - <[0] expired tickets removed.>
<30>Jul 20 15:43:14 localhost.localdomain cas[1017]: #033[m#033[33m2022-07-20 15:43:14,615 WARN [org.apereo.cas.services.RegisteredServiceAccessStrategyUtils] - <Unauthorized Service Access. Service [applianceStatusConnectionTest] is not found in service registry.>
<30>Jul 20 15:43:24 localhost.localdomain cas[1017]: #033[m#033[33m2022-07-20 15:43:24,640 WARN [org.apereo.cas.services.RegisteredServiceAccessStrategyUtils] - <Unauthorized Service Access. Service [applianceStatusConnectionTest] is not found in service registry.>

TLS-secured TCP log forwarding

Configuring log forwarding

If you want to, for example, configure CAS logs to also go to the SIEM, you create new configuration 09_cas_fwd.conf with following contents:

[root@localhost rsyslog.d]# cat 09_cas_fwd.conf

global(DefaultNetstreamDriverCAFile="/etc/pki/rsyslog/ncat.crt")	# location of CA certificate of your SIEM server
if $programname == "cas" then {
	action (
		type="omfwd" protocol="tcp" target="10.0.1.247" port="514"
		StreamDriver="gtls"			# GnuTLS for encrypted transport
		StreamDriverMode="1"			# TLS-only mode
		StreamDriverAuthMode="x509/name"	# cert validation + server's cert subject name auth
#		StreamDriverPermittedPeers="localhost"	# if not set, CN/SANs are compared to target
#		StreamDriver.PermitExpiredCerts="warn"  # on / off / warn (allow but log a warning)

		queue.type="LinkedList"
		queue.filename="cas-sending-q"		# queue name on the disk, adjust for each queue
		queue.size="10000"			# about 1kb per message -> 100MB maxsize
		queue.maxDiskSpace="1g"			# 1GB stored on disk at most
#		queue.discardSeverity="8"		# discard messages of any severity. rsyslog default.
		queue.checkpointInterval="1000"		# write queue checkpoint every 1000 messages
		queue.maxFileSize="64m"			# queue fragment file size
		queue.saveOnShutdown="on"		# writeback to disk on queue shutdown

		action.resumeRetryCount="-1"		# do not fail this action. try indefinitely
		action.reportSuspension="on"		# log wehn this action is suspended/resumed
	)
}

Key points here are:

This configuration is placed in 09_cas_fwd.conf which goes before 10_cas.conf. If it was loaded after 10_cas.conf, it would not work because the 10_cas.conf contains a stop clause.
The configuration in 09_cas_fwd.conf does not contain a stop clause. Logs are sent over the network according to 09_cas_fwd.conf but also further processed which also directs them into the local logfile as is specified in the 10_cas.conf.
This configuration uses disk-backed in-memory rsyslog queue. Maximum memory consumption is 100MB and maximum disk consumption is 1GB. When the disk cache overflows, rsyslog starts to drop messages.
The TLS transport is used, rsyslog checks the certificate of the server when making a connection. It also checks the certificate CN and/or SANs against the StreamDriverPermittedPeers list. If StreamDriverPermittedPeers is not specified, rsyslog performs the check against the target.
If you want to forwarding just CAS audit logs, you need to add the AND $msg contains 'Slf4jLoggingAuditTrailManager' part to filter line. This filter works only for CAS logs.

You can modify the log forwarding configuration completely to your liking but you must not edit configuration files that come with the IAM appliance.

Make sure the rsyslog-gnutls RPM package is installed.

After you configure the rsyslog, restart it for changes to take effect.

[root@localhost ~]# systemctl restart rsyslog

Testing

Rsyslog now starts to send logs over the network. If you configured standard UDP on port 514, you can easily test with running ncat (part of nmap package) on the target machine. Example of forwarded logs (newlines added for clarity).

[root@siem ~]# ncat -klvnp 514 --ssl-key ncat.key --ssl-cert ncat.crt
<30>Jul 20 15:43:12 localhost.localdomain cas[1017]: #033[m#033[32m2022-07-20 15:43:12,399 INFO [org.apereo.cas.ticket.registry.DefaultTicketRegistryCleaner] - <[0] expired tickets removed.>
<30>Jul 20 15:43:14 localhost.localdomain cas[1017]: #033[m#033[33m2022-07-20 15:43:14,615 WARN [org.apereo.cas.services.RegisteredServiceAccessStrategyUtils] - <Unauthorized Service Access. Service [applianceStatusConnectionTest] is not found in service registry.>
<30>Jul 20 15:43:24 localhost.localdomain cas[1017]: #033[m#033[33m2022-07-20 15:43:24,640 WARN [org.apereo.cas.services.RegisteredServiceAccessStrategyUtils] - <Unauthorized Service Access. Service [applianceStatusConnectionTest] is not found in service registry.>

Forwarding only the audit logs

Create the configuration in the same way as you would do in udp log forwarding or tcp log forwarding and change the line which contains the message filtering statement.

For CzechIdM, it should read:

if $programname == "czechidm" AND $msg contains "AUDIT." then {
...

For CAS, it should read:

if $programname == "cas" AND $msg contains "Slf4jLoggingAuditTrailManager" then {
...

After you configure the rsyslog, restart it for changes to take effect.

[root@localhost ~]# systemctl restart rsyslog