iWatch my logs in Splunk....do you?

iWatch is a program written in perl that performs real-time filesystem monitoring and requires inotify support for Linux-based kernels (Linux Kernel >= 2.6.13).  Its purpose is to monitor changes in a specific directory, file or even recursively monitor a directory and perform event notification immediately after a change.  This program is somewhat similar to the open source versions of Tripwire, OSSEC and AIDE however, it is more simplistic in nature and can easily be tied into your central syslog monitoring solution.

Activity
Configure iWatch to monitor some critical files on a local Ubuntu Linux server and report changes to syslog.  Then configure Splunk (the standard Search App) to monitor the local syslog file and modify syslog-event transformations to display iWatch specific fields.

• It will be assumed that Ubuntu is already installed and operating 
• Local Ubuntu box is configured to have syslog messages forwarded to /var/log/syslog 
• Splunk 4.1.x is already configured and the default Search app is available

Step 1:

1. Download inotify support and iwatch for Ubuntu (I used archive.offensive-security.com as my source repository)

• # apt-get install inotify-tools

• # apt-get install iwatch

Step 2: Configure the local syslog file index in Splunk

• Log into the Splunk Manager web interface Manager --> Data inputs --> Files & Directories --> Add New
  
• Click the Index a file on the Splunk server 
• Use /var/log/syslog as the path
• Select From list as the sourcetype, then select syslog as the source type
  
• Select main as the index, unless you know what you are doing
• Select Follow tail (so Splunk performs a tail -f on the file and reads in new events after you create the index)
• The click Save 

Step 3: Modify the syslog-event transformations file:
In the splunk web interface, now go to Manager --> Fields --> Field transformations --> syslog-extractions
Modify the fields as follows, then click the Save button when completed.

Regular Expression:
\s([^\s\[]+)(?:\[(\d+)\])?:\s(.*?)\s(.*?)\s

Key:

_raw

Event Format:

process::$1 pid::$2 iwatch_event::$3 iwatch_file::$4

Step 4: Start local iWatch process
-s (log events to syslog)
-v (verbose)
-r (recursive monitor directories/files)
-e (monitor all file modifications/changes)

(run in foreground)

# iwatch -s -v -r -e all_events /etc /bin

or
(run in background)
# iwatch -s -v -r -e all_events /etc/bin &


Step 5: create, delete some files  (in bash shell as root)

# cd /etc; i=0; while [ $i -le 10 ]; do touch file.$i; i=`expr $i + 1`; done

# cd /etc; j=0; while [ $j -le 10 ]; do rm file.$j; j=`expr $j + 1`; done

(view contents of sylog and make sure we have some events)

# cat /var/log/syslog

Step 6: Search for the new iWatch events in Splunk
Using the Splunk Search App, search for your new events using the following regex:

sourcetype="syslog" process="iWatch"

You should see "iwatch_event" and "iwatch_file" under the interesting field column on the left.  Click on those fields and search for your specific events.  Now you have the ability to search and build reports on file modifications/changes to your environment!!!  I would recommend reading more on iWatch to see how it can best be configured to work in your environment.  The example above should just get you started.

Happy Spelunking!!!


References
http://iwatch.sourceforge.net/index.html