Tag Archives: logging

NXlog – Parsing Squid access logs to json

For those of you using nxlog agents to parse logs here is a working squid access.log configuration:

<Extension squid_parse_action>
      Module xm_csv
      Fields $HTTPMethod, $HTTPResponseStatus
      FieldTypes string, string
      Delimiter '/'
      EscapeControl FALSE
      UndefValue -
</Extension>

<Extension squid_parse_hierarchy>
      Module xm_csv
      Fields $action, $dst_ip
      FieldTypes string, string
      Delimiter '/'
      EscapeControl FALSE
      UndefValue -
</Extension>

<Input in_file_squid_access_log>
      Module im_file
      File "/var/log/squid/access.log"
      SavePos TRUE
      ReadFromLast TRUE
      Exec if $raw_event =~ /^(\S+)\s+(\S+) (\S+) (\S+) (\S+) (\S+) (\S+) (\S+) (\S+) (\S+)/ \
      { \
           $epochtimetmp = $1; \
           $duration = $2; \
           $src_ip = $3; \
           squid_parse_action->parse_csv($4);\
           $FileSize = $5; \
           $HTTPMethod = $6; \
           $HTTPURL = $7; \
           $ident = $8; \
           squid_parse_hierarchy->parse_csv($9); \
           $contenttype = $10; \
           if $epochtimetmp =~ s/\.//g; \
           $epochtime=integer($epochtimetmp)*1000;\
           $EventTime = datetime($epochtime); \
           $type = "squid_access_log"; \
           $Hostname = hostname(); \
           $MessageSourceAddress = hostname(); \
           $Message = $raw_event; \
           to_json();\
      } \
      else \
      {\
           $Hostname = hostname(); \
           $MessageSourceAddress = hostname(); \
           $Message=$raw_event; \
           to_json();\
      }
</Input>

The timestamp in the squid log file is in unixtime and is in seconds with milliseconds after the decimal place. The datetime function converts unix/epoch time into datetime but expects an integer containing microseconds. For this reason the function removes the decimal place and multiplies the result by 1000 to get the number of microseconds.

Tagged ,

Snare plugin not working on AlienVault OSSIM

Alienvault OSSIM is a great open-source product but I recently struggled to get my Snare logs to show up in the Security Events (SIEM) viewer.

First of all follow the steps in the Snare setup guide.

Now that you’ve setup snare and enabled the plugin you should see the plugin running when you go to Configuration -> Collection -> Sensors -> Click on the SIEM host. The SnareWindows plugin should show up as ENABLED and UP.

Check the Security Events for snare logs. They should look like:

Snare: A service was successfully sent a start/stop control

If you don’t see any then try this:

1. SSH to the server

2. As root type

vi /etc/ossim/agent/plugins/snare.cfg

3. Find the line that says:

location=/var/log/snare.log

4. Comment it out by putting a # at the start of the line so it looks like this:

#location=/var/log/snare.log

5. Now copy and paste the line and change the snare.log part to say messages so it looks like this:

#location=/var/log/snare.log
location=/var/log/messages

6. Restart the ossim-agent

/etc/init.d/ossim-agent restart

Now go back and check your Security Events view. If your problem was the same as mine then your snare logs should now start appearing.

Update

Perhaps the best way to reduce the CPU load caused by the agent parsing is to split the syslogs into separate log files using rsyslog. It needs a bit of planning in advance. Configure your snare installs to send their syslog messages to a particular facility, e.g. Local3. You can then setup rsyslog to split these into a new snare log.

Edit /etc/rsyslog.conf and scroll to the bottom and add in something like:

#Windows logs
if $syslogfacility-text == 'local3' then /var/log/snare.log

and restart the rsyslogd (/etc/init.d/rsyslog restart). Tail the /var/log/snare.log file to see if the logs are coming into the file correctly.

Now edit /etc/ossim/agent/plugins/snare.cfg and change the location line to point to this new log file:

location=/var/log/snare.log

Now edit the agent configuration file to include the plugin (/etc/ossim/agent/config.cfg). Scroll to the [plugins] section and check if the snare.cfg is listed. If it isn’t add it in:

snare-monitor=/etc/ossim/agent/plugins/snare.cfg

now restart the agent service

/etc/init.d/ossim-agent restart

and check back in the GUI.

If you still get nothing in the GUI it could be that the logs that are coming into the log file don’t match  the regex’s in the snare.cfg. To test this take a small sample of your logs and create a test log file, e.g tail -100 /var/log/snare.log > /var/log/snare-test.log. Now use the following to test the regex’s:

/usr/share/ossim/scripts/regexp.py /var/log/snare-test.log /etc/ossim/agent/plugins/snare.cfg q

You should see some lines being matched.

Some logs to check for errors:

/var/log/ossim/agent.log
/var/log/ossim/server.log

 

 

 

 

 

Tagged , , , , ,