Let's look at an example. A recent investigation was focusing on what sites were hosting exploit kit materials. We had a starting point in that we knew a given IP address was hosting an exploit kit, and we wanted to see what other sites hosted related material. The initial search was:
+dstip:82.192.87.28 groupby:site
freshnewstoday.org
893g.in
panam-airline.ru
www.fruitcoctails.com
hife.in
hub7.yourgreatfind.info
So, the question was, what other IP's were also hosting these site names? This will lead us to other hostile IP's which could be actionable intelligence for blocking at the firewall. For this, we use the new subsearch() transform:
+dstip:82.192.87.28 groupby:site | subsearch(class:url groupby:dstip)
Which yields all of the unique IP's:
82.192.87.28
218.85.136.44
199.19.53.1
208.67.222.222
8.28.16.201
123.108.111.67
199.254.31.1
74.54.56.23
199.249.112.1
199.19.54.1
184.173.149.222
192.33.14.30
216.69.185.8
8.28.16.203
125.19.40.90
Now, let's say we are only concerned with the non-US IP addresses. We can apply one of the previously existing transforms to do a whois lookup followed by a filter:
+dstip:82.192.87.28 groupby:site | subsearch(class:url groupby:dstip) | whois | filter(cc,us)
123.108.111.67 org=PANGNET cc=HK name=Pang International Limited descr=Pang International Limited
125.19.40.90 org=EXL-9803-nod cc=IN name=EXL EXL, A-48 SECTOR-58 NOIDA Uttar Pradesh India Contact Person: Mr.Neeraj Email: Neeraj.Jain@exlservice.com Phone:981809784 descr=EXL EXL, A-48 SECTOR-58 NOIDA Uttar Pradesh India Contact Person: Mr.Neeraj Email: Neeraj.Jain@exlservice.com Phone:981809784
218.85.136.44 org=CHINANET-FJ cc=CN name=CHINANET Fujian province network Data Communication Division China Telecom descr=CHINANET Fujian province network Data Communication Division China Telecom
82.192.87.28 org=LEASEWEB cc=NL name=LeaseWeb P.O. Box 93054 1090BB AMSTERDAM Netherlands www.leaseweb.com descr=LeaseWeb P.O. Box 93054 1090BB AMSTERDAM Netherlands www.leaseweb.com
Here something interesting has happened: since our last transform was ending in a groupby which summarizes the dstip field, ELSA has rolled up all of the whois transform fields and made them an opaque string for presentation after filtering for the non-US addresses. If we want to continue with another transform, it will not do the string summary so that it can pass the results to the next transform in the native format.
So, let's say that we don't want to see the results for 82.192.87.28 because we already know about that one. We can add another subsearch to filter that out
+dstip:82.192.87.28 groupby:site | subsearch(class:url groupby:dstip) | whois | filter(cc,us) | subsearch(class:url -82.192.87.28 groupby:site)
We get back:
apps.db.ripe.net
whois.arin.net
Obviously, these are hits from our node's lookups and not relevant. Why did these come up? The subsearch actually ran this query:
class:url -82.192.87.28 groupby:site +(82.192.87.28 218.85.136.44 193.0.6.142 10.56.64.145 125.19.40.90 123.108.111.67)
So, any log in class URL that contains any of these terms (except 82.192.87.28 because it was negated) will match. That means that an URL like
GET whois.arin.net/rest/ip/218.85.136.44
will be found in this match. We don't want that. Luckily, there is a way to further filter this by adding on a field name (dstip) as the second argument to subsearch like this:
+dstip:82.192.87.28 groupby:site | subsearch(class:url groupby:dstip) | whois | filter(cc,us) | subsearch(class:url -82.192.87.28 groupby:site,dstip)
Now we get no results back, which means that there were no hits to any clients visiting sites on the other IP addresses because none of the IP's in the previous search showed up as a dstip in class URL.
Let's look at another example. Find all clients who were attacked by an exploit kit like Blackhole:
+sig_msg:"exploit kit" groupby:srcip
Within those results, find the dstip's that any Java web requests went to:
| subsearch(+user_agent:java groupby:dstip)
Perform a whois transform and only include Russian sites:
| whois | grep(cc,ru)
Now find all sites hosted on these Russian IP's:
| subsearch(class:url groupby:site,dstip)
Found some more bad sites!
cat.ms.softspb.com
svyaz62.findhere.org
svyaz61.findhere.org
vlhrusbi.athersite.com
sjpvmakd.3d-game.com
www.spbtraveler1.com
www.spbtraveler2.com
91.196.216.152
www.spbtraveler3.com
svyaz54.findhere.org
Correlation is a powerful tool for combining multiple simple notions to create a very sophisticated question. The ELSA utility transforms, like whois, provide a good way of whittling down large amounts of data into something interesting for use in alerting and reporting. It is sometimes the only way to tease out bits of data that would otherwise blend in with the vast amount of similar logs.
