We are very fortunate where I work to have Splunk. It’s an incredibly powerful indexing tool that can “eat all your IT data” and report on it in many different ways. We mostly use it to do simple searches for troubleshooting, but we’re always building more expertise as time permits.
Splunk is set up to index syslog messages very nicely by default. It takes each syslog message and intelligently recognizes the date/time stamp, then “extracts” all the fields and names them things like “host”, “eventtype”, “event_desc”, “error_code”, “log_level”, and so on. This post focuses on the “host” field, which is the IP address of the end device (router, switch, firewall, etc).
In our environment, we send all our syslogs to a Linux server running a free open-source tool called syslog-ng. With it, we do two things: (1) save a copy of each syslog message on the local server in a flat text file named for the source IP address where it came from, and (2) forward a copy to our Splunk indexing server using TCP port 9998.
For a while I’ve noticed that our Splunk lists all syslog messages with a “host” field that is the IP of the syslog-ng server. I was able to do some research this morning and “fixed” this so now all the syslog-ng forwarded messages have their host field set to the source IP address of their original sending device (router/switch/firewall).
Here’s how I did it:
1. Created props.conf file in /san/splunk/etc/system/local with the following contents
TRANSFORMS = syslog-header-stripper-ts-host syslog-host
2. Then restarted splunk with this command:
service splunk restart
Information sources I used: