Tag Archives: shell

Keeping OSSIM DB tables in check

In every OSSIM install I have done I’ve found the built in system for keeping the database size to manageable level doesn’t work very well. Eventually several tables gobble up all the disk space and fill the mysql partition. To make matters worse the default install uses a single ibdata file which doesn’t get released when you delete entries from the table.

So, I wrote this script to periodically check several problem tables and remove the oldest entries when they get to a certain size. What this size is depends on your environment. The problem tables I found to be were:

  • alienvault.extra_data
  • alienvault_siem.extra_data
  • alienvault.host_qualification
  • alienvault_siem.acid_event

I’ve just copied and pasted the same code 4 times as I was feeling lazy. You can find the mysql root password in /etc/ossim/ossim_setup.conf. Input that into the MYSQL_PASS variable. By default to script looks for when the number of rows in the table is greated than the THRESHOLD_ROWS value of 5000000. It will remove the number of rows that are more than this up to 5000000 in one go, so you should probably set this to run frequently, say hourly in a busy system.


#!/bin/bash
MYSQL_USER="root"
MYSQL_PASS="123456789"
THRESHOLD_ROWS=5000000
TARGET_DB="alienvault"
TARGET_TABLE="extra_data"
TARGET_COL_ORDER="event_id"
ROWS_PER_RUN=5000000

echo "Getting row count from table $TARGET_DB.$TARGET_TABLE"
numrows=`mysql -u $MYSQL_USER --password=$MYSQL_PASS -e "select count($TARGET_COL_ORDER) from $TARGET_DB.$TARGET_TABLE" | grep -v count | grep -v "-"`

echo "Got $numrows rows from table $TARGET_DB.$TARGET_TABLE"

if [ $numrows -gt $THRESHOLD_ROWS ]
then
        let "rowstodelete = $numrows - $THRESHOLD_ROWS"
        if [ $rowstodelete -gt $ROWS_PER_RUN ]
        then
                echo "Deleting $ROWS_PER_RUN from table $TARGET_DB.$TARGET_TABLE"
                result=`mysql -u $MYSQL_USER --password=$MYSQL_PASS -e "delete from $TARGET_DB.$TARGET_TABLE order by $TARGET_COL_ORDER ASC LIMIT $ROWS_PER_RUN"`
        else
                echo "Deleting  $rowstodelete from table $TARGET_DB.$TARGET_TABLE"
                result=`mysql -u $MYSQL_USER --password=$MYSQL_PASS -e "delete from $TARGET_DB.$TARGET_TABLE order by $TARGET_COL_ORDER ASC LIMIT $rowstodelete"`
        fi
        echo "Result=$result (blank is good)"
else
        echo "Number of rows = $numrows which not more than $THRESHOLD_ROWS so nothing to do."
fi

TARGET_DB="alienvault_siem"
TARGET_TABLE="extra_data"
TARGET_COL_ORDER="event_id"

echo "Getting row count from table $TARGET_DB.$TARGET_TABLE"
numrows=`mysql -u $MYSQL_USER --password=$MYSQL_PASS -e "select count($TARGET_COL_ORDER) from $TARGET_DB.$TARGET_TABLE" | grep -v count | grep -v "-"`

echo "Got $numrows rows from table $TARGET_DB.$TARGET_TABLE"

if [ $numrows -gt $THRESHOLD_ROWS ]
then
        let "rowstodelete = $numrows - $THRESHOLD_ROWS"
        if [ $rowstodelete -gt $ROWS_PER_RUN ]
        then
                echo "Deleting $ROWS_PER_RUN from table $TARGET_DB.$TARGET_TABLE"
                result=`mysql -u $MYSQL_USER --password=$MYSQL_PASS -e "delete from $TARGET_DB.$TARGET_TABLE order by $TARGET_COL_ORDER ASC LIMIT $ROWS_PER_RUN"`
        else
                echo "Deleting  $rowstodelete from table $TARGET_DB.$TARGET_TABLE"
                result=`mysql -u $MYSQL_USER --password=$MYSQL_PASS -e "delete from $TARGET_DB.$TARGET_TABLE order by $TARGET_COL_ORDER ASC LIMIT $rowstodelete"`
        fi
        echo "Result=$result (blank is good)"
else
        echo "Number of rows = $numrows which not more than $THRESHOLD_ROWS so nothing to do."
fi

TARGET_DB="alienvault"
TARGET_TABLE="host_qualification"
TARGET_COL_ORDER="hex(host_id)"

echo "Getting row count from table $TARGET_DB.$TARGET_TABLE"
numrows=`mysql -u $MYSQL_USER --password=$MYSQL_PASS -e "select count(host_id) from $TARGET_DB.$TARGET_TABLE" | grep -v count | grep -v "-"`

echo "Got $numrows rows from table $TARGET_DB.$TARGET_TABLE"

if [ $numrows -gt $THRESHOLD_ROWS ]
then
        let "rowstodelete = $numrows - $THRESHOLD_ROWS"
        if [ $rowstodelete -gt $ROWS_PER_RUN ]
        then
                echo "Deleting $ROWS_PER_RUN from table $TARGET_DB.$TARGET_TABLE"
                result=`mysql -u $MYSQL_USER --password=$MYSQL_PASS -e "delete from $TARGET_DB.$TARGET_TABLE order by $TARGET_COL_ORDER ASC LIMIT $ROWS_PER_RUN"`
        else
                echo "Deleting  $rowstodelete from table $TARGET_DB.$TARGET_TABLE"
                result=`mysql -u $MYSQL_USER --password=$MYSQL_PASS -e "delete from $TARGET_DB.$TARGET_TABLE order by $TARGET_COL_ORDER ASC LIMIT $rowstodelete"`
        fi
        echo "Result=$result (blank is good)"
else
        echo "Number of rows = $numrows which not more than $THRESHOLD_ROWS so nothing to do."
fi

TARGET_DB="alienvault_siem"
TARGET_TABLE="acid_event"
TARGET_COL_ORDER="timestamp"

echo "Getting row count from table $TARGET_DB.$TARGET_TABLE"
numrows=`mysql -u $MYSQL_USER --password=$MYSQL_PASS -e "select count($TARGET_COL_ORDER) from $TARGET_DB.$TARGET_TABLE" | grep -v count | grep -v "-"`

echo "Got $numrows rows from table $TARGET_DB.$TARGET_TABLE"

if [ $numrows -gt $THRESHOLD_ROWS ]
then
        let "rowstodelete = $numrows - $THRESHOLD_ROWS"
        if [ $rowstodelete -gt $ROWS_PER_RUN ]
        then
                echo "Deleting $ROWS_PER_RUN from table $TARGET_DB.$TARGET_TABLE"
                result=`mysql -u $MYSQL_USER --password=$MYSQL_PASS -e "delete from $TARGET_DB.$TARGET_TABLE order by $TARGET_COL_ORDER ASC LIMIT $ROWS_PER_RUN"`
        else
                echo "Deleting  $rowstodelete from table $TARGET_DB.$TARGET_TABLE"
                result=`mysql -u $MYSQL_USER --password=$MYSQL_PASS -e "delete from $TARGET_DB.$TARGET_TABLE order by $TARGET_COL_ORDER ASC LIMIT $rowstodelete"`
        fi
        echo "Result=$result (blank is good)"
else
        echo "Number of rows = $numrows which not more than $THRESHOLD_ROWS so nothing to do."
fi

You can also download the script here.

Tagged ,

Mimicking a multi-stage update server for RHEL servers on Classic

One of Redhat Satellite’s functions is update release management. Unfortunately it also quite expensive. If you have classic you can mimic the update features (albeit not the security updates) through some scripting.

This solution uses a server which is registered with Redhat with a Classic subscription so it’s not quite free. The server is able to reposync all the updates for its architecture and major release version (e.g. RHEL 5 x86_64).

1. Register the server you want to use as the reposync server on Redhat Classic and assign the correct channel.

2. Using the following command to create a new repo:

reposync -p /mypath/mirror/ --repoid=rhel-x86_64-server-5 -l -n

This syncs the whole of Redhat’s rhel-x86_64-server-5 repository to the /mypath/mymirror folder. You can change these paths and repoid to however you please. The -n switch means that only the newest files are synced each run so you can schedule this in your crontab to run regularly.

3. Now you want to create a point in time snapshot of all the updates contained in the mirror at that point in time. To do that you can use lndir. Here is my rhn_create_snapshot.sh script:

#!/bin/sh

baseRepoPath=/mypath

function usage()
{
 echo "This script creates a new snapshot folder for yum updates"
 echo
 echo "rhn-create-snapshot.sh MYGROUP"
 echo "-h --help Show this help"
 echo "--environment=MYGROUP Set the environment to create the repo for"
 echo
}
while [ "$1" != "" ]; do
 PARAM=`echo $1 | awk -F= '{print $1}'`
 VALUE=`echo $1 | awk -F= '{print $2}'`
 case $PARAM in
 -h | --help)
 usage
 exit
 ;;
 --environment)
 currEnvironment=$VALUE
 ;;
 *)
 echo "ERROR: unknown parameter \"$PARAM\""
 usage
 exit 1
 ;;
 esac
 shift
done
if [ "$currEnvironment" == "" ]; then
 echo "Environment not set properly. Exiting."
 usage
 exit 1
fi
currDate=$(date +%Y%m%d-%k%m)

echo "Creating yum snapshot for environment $currEnvironment in $baseRepoPath/snapshots/rhel-x86_64-server-5/$currDate/packages"

echo "Creating new folder for the snapshot to go in"
mkdir -p $baseRepoPath/snapshots/rhel-x86_64-server-5/$currDate/packages

echo "Creating new folder for the repodata to go in"
mkdir -p $baseRepoPath/snapshots/rhel-x86_64-server-5/$currDate/repodata

echo "Creating a directory of links in the snapshot folder"
lndir $baseRepoPath/mirror/rhel-x86_64-server-5/getPackage/ $baseRepoPath/snapshots/rhel-x86_64-server-5/$currDate/packages

echo "Creating a repo for the client to read"
createrepo $baseRepoPath/snapshots/rhel-x86_64-server-5/$currDate

echo "Creating a link for the client to the snapshot directory"
if [ -e $baseRepoPath/updates/$currEnvironment ]; then
 rm -f $baseRepoPath/updates/$currEnvironment
fi

ln -s $baseRepoPath/snapshots/rhel-x86_64-server-5/$currDate $baseRepoPath/updates/$currEnvironment

echo "All done"
echo "Please use the following yum config in your yum.conf:"
echo
echo "[update]"
echo "gpgcheck=0"
echo "name=Red Hat Linux Updates"
echo "baseurl=http://<servername>/updates/$currEnvironment"
echo

If you run

sh ./rhn_create_snapshot.sh MYGROUP

it creates a folder structure like this:

mypath
       mirror
             rhel-x86_64-server-5
                    getPackage
                            <list of reposync'd packages>
       updates
            MYGROUP -> /mypath/snapshots/rhel-x86_64-server-5/2013-12-11-1104
       snapshots
            rhel-x86_64-server-5
                    2013-12-11-1104
                            packages
                                   <directory of symbolic links to /mypath/mirror/rhel-x86_64-server-5/getPackage/<filename>
                            repodata

Each time the rhn_create_snapshot.sh script is run is creates a new dated snapshot directory under /mypath/snapshots/rhel-x86_64-server-5 containing a packages folder with a directory full of symbolic links of the files that were in /mypath/mirror/rhel-x86_64-server-5/getPackage at the time it was run. A repo is then created on this list of files. The only space the snapshot takes up is for the repodata files, which is about 200MB in size.

So now you can roll patches through groups of servers by setting the baseurl in their yum.conf to point to different updates/MYGROUP. For example:

TEST -> /mypath/snapshots/rhel-x86_64-server-5/2013-12-11-1104
UAT -> /mypath/snapshots/rhel-x86_64-server-5/2013-09-11-1001
PROD -> /mypath/snapshots/rhel-x86_64-server-5/2013-17-05-1630

The PROD group could be getting a different set of packages when yum update is run than UAT or LIVE. All you need to do to release TEST packages to UAT is to change the symbolic link. Using the above example:

rm -f /mypath/updates/UAT
ln -s /mypath/snapshots/rhel-x86_64-server-5/2013-12-11-1104 /mypath/updates/UAT

which now gives:

TEST -> /mypath/snapshots/rhel-x86_64-server-5/2013-12-11-1104
UAT -> /mypath/snapshots/rhel-x86_64-server-5/2013-12-11-1104
PROD -> /mypath/snapshots/rhel-x86_64-server-5/2013-17-05-1630

Now going to UAT machine and running yum update should list new updates. You could no doubt script that if you wanted.

Tagged , , , , , ,

Script to control NVidia GPU fan under linux using nvclock

This is a modified script I found at  http://www.linuxjournal.com/content/nvidia-fan-speed-revisited. The script didn’t work for me so I edited a little to work under shell.

This script tries to keep the GPU temperature within 1C of 54C. The max clock speed is 100% and the lowest 20%. It clocks up and down at 5% at a time at intervals of 5 seconds until the target temperature is reached.

You will need to install the proprietary driver and nvclock from your distribution’s repository.

I have added this script to my rc.local to run at boot-time in the back ground and I find it works well.

/bin/sh /$SCRIPTDIR/nvclock_fan.sh >> /var/log/messages 2>&1

The script:

#!/bin/bash
#
# Adjust fan speed automatically.
# This version by StuJordan
# Based on Version by DarkPhoinix
# Original script by Mitch Frazier

# Location of the nvclock program.
nvclock_bin=/usr/bin/nvclock

# Target temperature for video card.
target_temp=54

# Value used to calculate the temperature range (+/- target_temp).
target_range=1

# Time to wait before re-checking.
sleep_time=5

# Minimum fan speed.
min_fanspeed=20

# Fan speed increment.
adj_fanspeed=5

if [ "$1" ]; then target_temp=$1; fi

target_temp_low=$(expr $target_temp - $target_range)
target_temp_high=$(expr $target_temp + $target_range)

while true
do
    temp_val=$(echo $($nvclock_bin --info | grep -i 'GPU temperature' | cut -d ':' -f 2) | cut -d C -f 1)
#    pwm_val=$(echo $($nvclock_bin --info | grep -i 'PWM' | cut -d ':' -f 2) | cut -d " " -f 1)
     pwm_val=$(echo $($nvclock_bin --info | grep -i 'PWM' | cut -d ':' -f 2 | cut -d " " -f 2 | cut -d "%" -f 1 | cut -d "." -f 1))

    echo "Current temp is $temp_val. Current pwm is $pwm_val"
    echo "Target temp high is $target_temp_high and low is $target_temp_low"

    if [ $temp_val -gt $target_temp_high ]; then
	echo "Temperature too high"

	# Temperature above target, see if the fan has any more juice.

        if [ $pwm_val -lt 100 ]; then
            echo "Increasing GPU fan speed, temperature: $temp_val"
            pwm_val=$(expr $pwm_val + $adj_fanspeed)
            if [ $pwm_val -gt 100 ]; then pwm_val=100; fi
            $nvclock_bin -f --fanspeed $pwm_val
        fi
    elif [ $temp_val -lt $target_temp_low ]; then

# Temperature below target, lower the fan speed
# if we're not already at the minimum.

        if [ $pwm_val -gt $min_fanspeed ]; then
            echo "Decreasing GPU fan speed, temperature: $temp_val"
            pwm_val=$(expr $pwm_val - $adj_fanspeed)
            if [ $pwm_val -lt $min_fanspeed ]; then pwm_val=$min_fanspeed; fi
            $nvclock_bin -f --fanspeed $pwm_val
        fi
    fi
    sleep $sleep_time
done
Tagged , ,

Updating snort and openvas rules

Openvas and snort rules in Alienvault OSSIM are deployed as part of the updates. However, you can update them more frequently directly from the Openvas and Snort repositories.

Openvas Plugin Update Script

Most of this is directly from the Alienvault configuration guide, but in assorted places. Here’s the script to update the openvas rules:

#!/bin/sh 
openvas-nvt-sync --wget /etc/init.d/openvas-scanner restart 
perl /usr/share/ossim/scripts/vulnmeter/updateplugins.pl migrate

Save this as a .sh file (e.g. update-openvas-plugins.sh) and chmod to 700 with owner root

chmod 700 update-openvas-plugins.sh 
chown root.root update-openvas-plugins.sh

Then add to root’s crontab:

crontab -e

and add the following line:

0 3 * * 6 /bin/sh /scripts/update-openvas-plugins.sh

where this one runs weekly on Saturday at 3am. For more info on editing crontab see here.

Snort Plugin Update Script

Here’s the script to update snort:

#!/bin/sh
perl /usr/share/ossim/scripts/create_sidmap.pl /etc/snort/rules/
/etc/init.d/ossim-server restart

If the box is just a snort collector and doesn’t have the ossim-server running you’ll want to change that last line to read:

/etc/init.d/snort restart

or

/etc/init.d/snort_eth1 restart

Where eth1 is the interface snort is attached to.

Then edit crontab again and add in the line:

0 4 * * 6 /bin/sh /scripts/update-snort-rules.sh

This one runs every Saturday at 4am.

Tagged , , , , , , , , , ,