Thursday, March 8, 2012

Correlation in ELSA

In my previous post, I showed how complex the data sets in security have become.  In order to cope with this amount of heterogeneous records, powerful correlation capabilities are required.  I'm pleased to announce that as of today, basic correlation is now possible in ELSA through the new subsearch() transform.

Let's look at an example.  A recent investigation was focusing on what sites were hosting exploit kit materials.  We had a starting point in that we knew a given IP address was hosting an exploit kit, and we wanted to see what other sites hosted related material.  The initial search was:

+dstip:82.192.87.28 groupby:site
   
freshnewstoday.org
893g.in
panam-airline.ru
www.fruitcoctails.com
hife.in
hub7.yourgreatfind.info

So, the question was, what other IP's were also hosting these site names?  This will lead us to other hostile IP's which could be actionable intelligence for blocking at the firewall.  For this, we use the new subsearch() transform:

+dstip:82.192.87.28 groupby:site | subsearch(class:url groupby:dstip)

Which yields all of the unique IP's:

82.192.87.28
218.85.136.44
199.19.53.1
208.67.222.222
8.28.16.201
123.108.111.67
199.254.31.1
74.54.56.23
199.249.112.1
199.19.54.1
184.173.149.222
192.33.14.30
216.69.185.8
8.28.16.203
125.19.40.90

Now, let's say we are only concerned with the non-US IP addresses.  We can apply one of the previously existing transforms to do a whois lookup followed by a filter:

+dstip:82.192.87.28 groupby:site | subsearch(class:url groupby:dstip) | whois | filter(cc,us)

123.108.111.67 org=PANGNET cc=HK name=Pang International Limited descr=Pang International Limited
125.19.40.90 org=EXL-9803-nod cc=IN name=EXL EXL, A-48 SECTOR-58 NOIDA Uttar Pradesh India Contact Person: Mr.Neeraj Email: Neeraj.Jain@exlservice.com Phone:981809784 descr=EXL EXL, A-48 SECTOR-58 NOIDA Uttar Pradesh India Contact Person: Mr.Neeraj Email: Neeraj.Jain@exlservice.com Phone:981809784
218.85.136.44 org=CHINANET-FJ cc=CN name=CHINANET Fujian province network Data Communication Division China Telecom descr=CHINANET Fujian province network Data Communication Division China Telecom
82.192.87.28 org=LEASEWEB cc=NL name=LeaseWeb P.O. Box 93054 1090BB AMSTERDAM Netherlands www.leaseweb.com descr=LeaseWeb P.O. Box 93054 1090BB AMSTERDAM Netherlands www.leaseweb.com

Here something interesting has happened: since our last transform was ending in a groupby which summarizes the dstip field, ELSA has rolled up all of the whois transform fields and made them an opaque string for presentation after filtering for the non-US addresses.  If we want to continue with another transform, it will not do the string summary so that it can pass the results to the next transform in the native format.

So, let's say that we don't want to see the results for 82.192.87.28 because we already know about that one.  We can add another subsearch to filter that out

+dstip:82.192.87.28 groupby:site | subsearch(class:url groupby:dstip) | whois | filter(cc,us) | subsearch(class:url -82.192.87.28 groupby:site)

We get back:

apps.db.ripe.net
whois.arin.net

Obviously, these are hits from our node's lookups and not relevant.  Why did these come up?  The subsearch actually ran this query:

class:url -82.192.87.28 groupby:site +(82.192.87.28 218.85.136.44 193.0.6.142 10.56.64.145 125.19.40.90 123.108.111.67)

So, any log in class URL that contains any of these terms (except 82.192.87.28 because it was negated) will match.  That means that an URL like

GET whois.arin.net/rest/ip/218.85.136.44

will be found in this match.  We don't want that.  Luckily, there is a way to further filter this by adding on a field name (dstip) as the second argument to subsearch like this:

+dstip:82.192.87.28 groupby:site | subsearch(class:url groupby:dstip) | whois | filter(cc,us) | subsearch(class:url -82.192.87.28 groupby:site,dstip)

Now we get no results back, which means that there were no hits to any clients visiting sites on the other IP addresses because none of the IP's in the previous search showed up as a dstip in class URL.

Let's look at another example.  Find all clients who were attacked by an exploit kit like Blackhole:
+sig_msg:"exploit kit" groupby:srcip

Within those results, find the dstip's that any Java web requests went to:
| subsearch(+user_agent:java groupby:dstip)

Perform a whois transform and only include Russian sites:
| whois | grep(cc,ru)

 Now find all sites hosted on these Russian IP's:
| subsearch(class:url groupby:site,dstip)

Found some more bad sites!

www.spbtraveler.com
cat.ms.softspb.com
svyaz62.findhere.org
svyaz61.findhere.org
vlhrusbi.athersite.com
sjpvmakd.3d-game.com
www.spbtraveler1.com
www.spbtraveler2.com
91.196.216.152
www.spbtraveler3.com
svyaz54.findhere.org

Correlation is a powerful tool for combining multiple simple notions to create a very sophisticated question.  The ELSA utility transforms, like whois, provide a good way of whittling down large amounts of data into something interesting for use in alerting and reporting.  It is sometimes the only way to tease out bits of data that would otherwise blend in with the vast amount of similar logs.

6 comments:

  1. Excellent post! Please keep the great work. I can see ELSA quickly becoming a must-have tool for every CSIRT!

    ReplyDelete
  2. Very interesting. I'm looking at performing a fairly complicated piece of correlation and would love to know if you think this can be done in Elsa.

    Say I wanted to see if the content of http://www.domain.com/image.png changed (in case it was being used for c2), would it be possible to do so?

    ReplyDelete
  3. If you are running Bro and logging to ELSA, then this could be done trivially by enabling MD5 calculation on the MIME type image/png, etc. The ELSA query would be:
    +www.domain.com +image.png groupby:md5 | has(2)
    That query would find any time there were two or more distinct MD5's calculated for that image.png file.

    Another tactic would be to monitor the bytes in the Bro HTTP or proxy log for the GET, or in some cases, if the remote server returns a "not modified" response. In both cases, you can use the same groupby/has combination to find the number of unique values.

    ReplyDelete
  4. Yes I am I'll check that out. Thankyou for your prompt reply!

    ReplyDelete
  5. I have read your blog its very attractive and impressive. I like it your blog.

    Java Online Training Java EE Online Training Java EE Online Training Java 8 online training Java 8 online training

    Java Online Training from India Java Online Training from India Core Java Training Online Core Java Training Online Java Training InstitutesJava Training Institutes

    ReplyDelete
  6. Bluehost is ultimately one of the best web-hosting company with plans for any hosting needs.

    ReplyDelete