New ELSA Log Parsers

ELSA is growing!

By popular demand, I've added a number of new parsers to the ELSA repertoire to support parsing fields from the following devices:
 - Fortinet (URL, traffic)
 - Checkpoint
 - Palo Alto (URL, traffic)
 - Barracuda (scan, receive, send)
 - OSSEC Windows logs (automatically appears as class Windows)

Parsed logs will have fields available for ad-hoc reporting and transforms.  For instance, now that srcip and dstip are available, the whois or cif transform can be used with these logs.

In order to use these classes, you will need to uncomment the class definitions in the node/conf/schema.sql file and manually do an INSERT of those classes.  This is to prevent those of us who do not have these devices from having a cluttered menu.  In the future, I'm hoping to have a much nicer way of administrating which classes are visible.

I've also added an updated syslog-ng.conf file which should handle Cisco device logs much better with the varied types of timestamps they send.

Lastly, I've updated the ELSA online documentation to show how to create your own parsers.

As always, if you have any questions or problems with ELSA, please let us know on the ELSA mailing list.

Hunting with ELSA Transforms

One of the talks at the Bro Workshop 2011 showed all of the amazing things you can do with Bro logs when manipulating them in Splunk by changing the content of the fields displayed. This reminded me that I've been planning on adding transforms to ELSA for some time, and I've finally got the first working version available!

Transforms

Transforms are post-search display filters that data is passed through to modify what is shown the user. For ELSA, this means that when the search engine is finished with its normal search, it will pass the list of results to a chain of filters strung together with pipes which can add fields or filter results. Currently, ELSA ships with the following transforms which add fields:

• Whois
• DNSDB (via account with dnsdb.isc.org)
• CIF (via local instance of Collective-Intelligence-Framework.googlecode.com)

and the following utility transforms:

• grep
• filter
• sum

Below is an example of a simple search which has Whois data added to each entry:

The query is a standard ELSA query, but using the pipe character (“|”) it is possible to send the results through the “whois” transform, which will add fields like country code “cc” and description “descr.” This can be crucial for analysts as they investigate because it saves them having to lookup IP addresses individually. As with everything in ELSA, the goal is to make delivering data as quick and effortless as possible to encourage investigation above and beyond typical searches. This makes for great hunting.

Hunting++

One easy way to start looking for anomalies in your IDS logs is to match up otherwise innocuous or “policy” signatures with anomalous locations or networks. In the screenshot below, you can see a search for executables downloaded from country code “ua” (Ukraine).

Using the info plugin (by clicking the link next to the log entry) shows the StreamDB output

in which you can clearly see that this is a download of “info.exe” caused by a Blackhole Exploit Kit due to the tell-tale “w.php?f=16&e=0” (which indicates that is is an update to an already-infected machine).

Setting Up Transforms

By default, whois will work with the default dummy configuration which declares local RFC1918 space as “MyOrg” as specified by your “known_orgs” entry in the elsa_web.conf under “transforms/whois.” The local IP will have the “customer” field added, which is specified by the “known_subnets” entry. ELSA ships with examples included, so you should only have to edit what already exists.

If you're upgrading ELSA, you may need to add a Perl module for faster result caching, as described below. To add it, run “sudo cpanm CHI::Driver::DBI” or ELSA will use memory-only caching, which will not persist across Apache restarts. This module is included in the install.sh.

Using Transforms

The main caveat to transforms is that you cannot search solely based on them. That's because they are applied after searches are finished, and can only be used to display additional information. This also means that the transforms are only applied to the number of results being returned. You may need to increase the number of results using the “limit” keyword to have the transforms applied to up to 1000 results. As an example, if you wanted to find all downloads with IP addresses related to Google, you would do this:

+sig_msg:exe limit:1000 | whois | grep(descr,google)

The main search is for 1000 alerts with “exe” in the message, which then has the whois fields added (cc, description, org), and then only records with a description matching “google” are returned. The grep and filter transforms are case-insensitive regular expressions on both the field name and the field value. If you want to search all fields for a given value, you can use the dot “.” instead of a field name, like this:

+sig_msg:exe limit:1000 | whois | grep(.,google)

All transform information is cached in the database to speed up result times. Generally, whois lookups are quite fast, but not nearly as fast as ELSA results usually are. However, if you run a search with a transform and then run it again with a different grep, filter, or sum, it should return almost instantly because it is using cached data.

Other Transforms

ELSA also ships with the dnsdb and cif transforms in addition to the whois transform. In order to use the dnsdb transform, you will need a working API key from ISC, which is free to non-profits. Contact them at dnsdb.isc.org for additional details. Below is a screenshot of what it looks like when you add on the passive DNS hosts to your search by passing the results through dnsdb:

It's verbose, but in combination with grep or filter, it can be very helpful.

The cif transform hooks into a local instance of the Collective Intelligence Framework which applies blacklist information as well as whois info to your results. Here's another hunting trip, this time looking for any IP's listed as type “malicious” from which there were downloads:

To use the CIF transform, you'll either need a local instance setup (highly recommended!) or a friendly org that is willing to give you an API key to their instance. In either case, edit the elsa_web.conf file to add the server name, IP, and apikey under “transforms/cif.”

Adding Transforms

Creating your own transforms is not entirely trivial, but isn't too hard. Take a look at the current ones in the web/lib/Transform directory, and with a lot of copy and paste, you should be able to hack your own together. If you get stuck, please drop us a note on the mailing list (http://groups.google.com/group/enterprise-log-search-and-archive).

ELSA Beta Available

After two months of solid coding, I'm proud to announce that Enterprise Log Search and Archive (ELSA) is available in beta quality and ships with an auto-installer for many Linux systems. The backend and query code has been rewritten from the ground up with many enhancements. Most importantly, query speed is an order of magnitude faster in many cases, and you can now report on string fields like any other field.  The query interface has been streamlined, along with other interface changes.

The rewrite focused on making the platform more robust by removing many moving parts and streamlining a lot of background processes. The result is agentless, fully independent log nodes which do not need to communicate with any other nodes. This allows for easy distribution of nodes in customer sites. In fact, the nodes can be accessed from as many different web frontends as you like, as they are nothing more than a MySQL query interface to external clients, so customers can have their own interface if they prefer. The web server now handles asynchronously parallelizing queries and aggregating the results, which has improved performance and reliability.

I've also added parsers and classes for the major Bro IDS logs, so ELSA is now a viable frontend for Bro. Below is a screenshot of a source IP address with a report by DNS hostname lookup. This search covered several billion logs and returned in less than a second.

There are many more features, and I will refer interested readers to the new documentation available. I'm very interested in any issues encountered during install and operation, so please let me know if you run into any issues. You may also file a bug report on the project page if you run into any problems.

Bro Quickstart: Cluster Edition

In my last post, I described the very basic commands you need to get a very minimal Bro IDS instance up and running for proof-of-concept purposes.  In this post, I'll show you how to take Bro a step further with Bro cluster.

Bro Cluster
There are several reasons for running Bro in a cluster.  A single Bro instance will be overwhelmed when it encounters more than about 80 Mb/sec of traffic.  You can run multiple Bro instances on the same machine (to take advantage of multiple cores) or multiple instances spread across distributed machines to cut the load each instance sees down to the 80 Mb/sec mark.

Another reason for using Bro cluster is the ease of managing Bro processes and logging.  There is a comprehensive log rotation framework integrated into Bro cluster, as well as seamless configuration changes on-the-fly.

Installation
This install quickstart will be written for Ubuntu Server 10.04 LTS, but aside from the prerequisite installation, the steps should be largely the same on any Linux distribution.  To run multiple Bro instances on the same local machine, we will use PF_RING's amazing ability to perform efficient, per-flow software pcap load-balancing.

# install prereqs
sudo apt-get install swig python-dev libmagic-dev libpcre3-dev libssl-dev cmake git-core subversion ruby-dev libgeoip-dev flex bison
# uninstall conflicting tcpdump
sudo apt-get remove tcpdump libpcap-0.8
cd ~/
# install PF_RING
svn export https://svn.ntop.org/svn/ntop/trunk/PF_RING/ pfring-svn
cd pfring-svn/kernel
make && sudo make install
cd ../userland/lib
./configure --prefix=/usr/local/pfring && make && sudo make install
cd ../libpcap-1.1.1-ring
./configure --prefix=/usr/local/pfring && make && sudo make install
echo "/usr/local/pfring/lib" >> /etc/ld.so.conf
cd ../tcpdump-4.1.1
./configure --prefix=/usr/local/pfring && make && sudo make install
# Add PF_RING to the ldconfig include list
echo "PATH=$PATH:/usr/local/pfring/bin:/usr/local/pfring/sbin" >> /etc/bash.bashrc
cd ~/
# Get and make Bro
mkdir brobuild && cd brobuild
git clone --recursive git://git.bro-ids.org/bro
BRODIR=/usr/local/bro-$(date +"%Y%m%d")
./configure --prefix=$BRODIR --with-pcap=/usr/local/pfring && cd build && make -j8 && sudo make install
sudo ln -s $BRODIR /usr/local/bro
cd /usr/local/bro


Now we need to edit the Bro config for your environment.  For the purposes of this quickstart, I will assume that you are monitoring a single interface, eth2.  Edit /usr/local/bro/etc/node.cfg to look like this (assuming your local host is 192.168.1.1 and you want to run 4 instances):

[manager]

type=manager

host=192.168.1.1

[proxy-0]

type=proxy

host=192.168.1.1

[worker-0]

type=worker

host=192.168.1.1

interface=eth2

lb_method=pf_ring

lb_procs=4

Now we need to edit share/bro/site/local.bro to add some options that I recommend to prevent some of the more verbose logs from filling up the drive.  Append the following text:


  event bro_init()

       {

       Log::disable_stream(HTTP::LOG);

        Log::disable_stream(Syslog::LOG);

        Log::disable_stream(Conn::LOG);

        Log::disable_stream(DNS::LOG);

        Log::disable_stream(Weird::LOG);

       }

Now we're all ready to start Bro cluster:

/usr/local/bro/bin/broctl

> install

> start

Finally, in an enterprise, you will want to get these logs into your log management or SIEM solution.  You can do this very easily out of the box in Ubuntu, which uses rsyslog by default.  Create /etc/rsyslog.d/60-bro.conf:

$ModLoad imfile #
$InputFileName /usr/local/bro/logs/current/ssl.log
$InputFileTag bro_ssl:
$InputFileStateFile stat-bro_ssl
$InputFileSeverity info
$InputFileFacility local7
$InputRunFileMonitor
$InputFileName /usr/local/bro/logs/current/smtp.log
$InputFileTag bro_smtp:
$InputFileStateFile stat-bro_smtp
$InputFileSeverity info
$InputFileFacility local7
$InputRunFileMonitor
$InputFileName /usr/local/bro/logs/current/smtp_entities.log
$InputFileTag bro_smtp_entities:
$InputFileStateFile stat-bro_smtp_entities
$InputFileSeverity info
$InputFileFacility local7
$InputRunFileMonitor
$InputFileName /usr/local/bro/logs/current/notice.log
$InputFileTag bro_notice:
$InputFileStateFile stat-bro_notice
$InputFileSeverity info
$InputFileFacility local7
$InputRunFileMonitor
$InputFileName /usr/local/bro/logs/current/ssh.log
$InputFileTag bro_ssh:
$InputFileStateFile stat-bro_ssh
$InputFileSeverity info
$InputFileFacility local7
$InputRunFileMonitor
$InputFileName /usr/local/bro/logs/current/ftp.log
$InputFileTag bro_ftp:
$InputFileStateFile stat-bro_ftp
$InputFileSeverity info
$InputFileFacility local7
$InputRunFileMonitor
# check for new lines every second
$InputFilePollingInterval 1
local7.* @central_syslog_server

Apply changes:

restart rsyslog

Monitoring SSL Connections with Bro: Quickstart

Updated (8/20/2011) based on more info from Seth.
 
Introduction
Bro (www.bro-ids.org) is an amazing suite of software which can do things that no other IDS on the planet can come close to.  In this post, I want to cover one such feature: SSL monitoring.  Bro has a true understanding of the SSL being used on your network and will efficiently process certificates on the wire for a variety of purposes.  Out of the box, Bro can very efficiently and accurately identify invalid and self-signed certificates, going so far as to actually walk the certificate chain using the certs that ship with Mozilla browsers for a true test.  In addition, Bro will extract all of the relevant details from certificates for logging purposes, which can provide a handy historical record of the sites and companies involved in SSL, which is the next best thing to performing proxy/MITM SSL inspection.

Installing Bro
This quickstart guide will show how to get up and running with Bro on Ubuntu.  I hope that most of the commands and tips will apply to other operating systems and Linux distros, but there will surely be some differences.

Begin by making sure we've got our prerequisites in order:
apt-get install git libssl-dev swig libmagic-dev libgeoip-dev
Grab the latest Bro from the git repository.  Beware, this is cutting edge code, and you may need to download the latest stable tarball from www.bro-ids.org if the git build fails:

git clone --recursive git://git.bro-ids.org/bro

Now you will have bro and auxiliary files in a directory named "bro."

cd bro

I have discovered that on some Linux distros (SuSE, for one), the version of CMake is less than  2.6.3 and so it needs to be downloaded from www.cmake.org and custom installed as Bro requires 2.6.3 or better.
(edited: "--enable-brov6" apparently has memory leaks right now.)
./configure --prefix=/usr/local/bro-git
There are a fair amount of options here, but the configure script does a pretty good job of finding out if you've got things installed already and adjusting accordingly.  Since we're looking to do SSL inspection, at a minimum, you'll need to make sure you've got the OpenSSL development libraries installed, which we've done above with apt-get.  If all goes, well, we do the make:

make && cd build && sudo make install

Now we will add a custom bro script which Seth Hall wrote which will print to STDOUT any SSL certificates which were created less than 30 days ago.

cd /usr/local/bro-git/share/bro/site/

vi young-ssl.bro

Paste in the following (edited: removed "@load protocols/ssl"):

event SSL::log_ssl(rec: SSL::Info)
       {
       # We have to check if there is a not_valid_before field because not
       # all SSL transactions actually exchange certificates (i.e. resumed session).
       if ( rec?$not_valid_before && rec$not_valid_before >= network_time() - 30 days &&
            rec$not_valid_before <= network_time() )
               {
               print fmt("%s is using a certificate that just became valid in the last 30 days (%T) (%s)",
                       rec$id$resp_h, rec$not_valid_before, rec$subject);
               }
       }

Now we activate it in the config:

echo "@load young-ssl" >> local.bro

Create some basic log directories for a test run:
mkdir /tmp/bro-logs
cd /tmp/bro-logs
Start bro (assuming we want to monitor eth1):
sudo /usr/local/bro-git/bin/bro -i eth1 local
Let it run for awhile, then have a look at the various logs created.  ssl.log will contain a list of all SSL certificates observed.  Here's an example:

# ts    uid    id.orig_h    id.orig_p    id.resp_h    id.resp_p    version    cipher    server_name    subject    not_valid_before    not_valid_after    validation_status
1313897881.475569    QUVGS5xx9ea    192.168.1.121    36804    199.59.148.87    443    TLSv10    TLS_DHE_RSA_WITH_AES_256_CBC_SHA    api.twitter.com    CN=api.twitter.com,OU=Twitter Platform,O=Twitter\, Inc.,L=San Francisco,ST=California,C=US    1274158800.000000    1337317199.000000    ok

So there you have it!  A fully functional Bro installation in just a few easy steps.  In a future post, I will show you have to get Bro output into various output collection mechanism like syslog and databases.

Running a load-balanced Snort in a PF_RING cluster

Even though Snort itself is single threaded, PF_RING has software load-balancing capabilities which will allow you to run it as if it were multi-threaded.  Here's the glossed-over version of the howto:

Note: By default, PF_RING ships with CLUSTER_LEN=8, which means only 8 processes can participate in a cluster.  If you have more than 8 cores and want to increase this amount, you will need to edit the source code for the PF_RING kernel module (<PF_RING_SRC>/kernel/linux/pf_ring.h and change #define CLUSTER_LEN 8 to 16 (or however many cores you have).  Then re-install the module (make && make install) and rmmod pf_ring && modprobe pf_ring to activate the new one.


1. Get PF_RING with the snort daq included
  svn co https://svn.ntop.org/svn/ntop/trunk/PF_RING/
2. Compile the daq (assuming PF_RING installed to /opt/PF_RING)
  ./configure --with-pic --with-libpcap-includes=/opt/PF_RING/include CFLAGS=-lpthread -lpfring -lpcap -D_GNU_SOURCE && make && make install
3. Add the following to your snort.conf:
config daq: pfring
config daq_dir: /usr/local/lib/daq
config daq_var: clusterid=44 (this can be any number < 255)
4. Start snort with a shell script wrapper like this (assuming you have 8 CPU's and you are sniffing eth2):
#!/bin/sh
for COUNTER in 0 1 2 3 4 5 6 7; do
mkdir /tmp/snort$COUNTER
kill $(cat /tmp/snort$COUNTER/snort_eth2.pid)
sleep 5;
/usr/local/snort/bin/snort -c /etc/snort/snort.conf --pid-path=/tmp/snort$COUNTER -l /tmp/snort$COUNTER --daq-var bindcpu=$COUNTER -D &
done
5. Profit

ELSA VMware Appliance Available

Peter over at Balabit has graciously offered a place to host a VM for ELSA.  You can download it at http://spike2.fa.gau.hu/~mcholste/elsa_vm.tar.gz .  It is a fully-functional ELSA installation running on Ubuntu 10.04 LTS.  It will start all necessary services to begin recording and viewing logs and provides a good way to see what ELSA is all about without a major time investment.  Please note that performance-wise, a VM will not be ideal, but it should be enough for interested readers to get a look at what ELSA can do.  The user name for the VM is "elsa" and pass is "biglog" .  I've included an SVN update script in the tarball now, so if you want to make sure the ELSA installation is current, you can run /usr/local/elsa/contrib/update_from_svn.sh and then execute "service elsa restart."  Please let me know if you run into any issues or have any comments!