Let's look at an example. A recent investigation was focusing on what sites were hosting exploit kit materials. We had a starting point in that we knew a given IP address was hosting an exploit kit, and we wanted to see what other sites hosted related material. The initial search was:
So, the question was, what other IP's were also hosting these site names? This will lead us to other hostile IP's which could be actionable intelligence for blocking at the firewall. For this, we use the new subsearch() transform:
+dstip:18.104.22.168 groupby:site | subsearch(class:url groupby:dstip)
Which yields all of the unique IP's:
Now, let's say we are only concerned with the non-US IP addresses. We can apply one of the previously existing transforms to do a whois lookup followed by a filter:
+dstip:22.214.171.124 groupby:site | subsearch(class:url groupby:dstip) | whois | filter(cc,us)
126.96.36.199 org=PANGNET cc=HK name=Pang International Limited descr=Pang International Limited
188.8.131.52 org=EXL-9803-nod cc=IN name=EXL EXL, A-48 SECTOR-58 NOIDA Uttar Pradesh India Contact Person: Mr.Neeraj Email: Neeraj.Jain@exlservice.com Phone:981809784 descr=EXL EXL, A-48 SECTOR-58 NOIDA Uttar Pradesh India Contact Person: Mr.Neeraj Email: Neeraj.Jain@exlservice.com Phone:981809784
184.108.40.206 org=CHINANET-FJ cc=CN name=CHINANET Fujian province network Data Communication Division China Telecom descr=CHINANET Fujian province network Data Communication Division China Telecom
220.127.116.11 org=LEASEWEB cc=NL name=LeaseWeb P.O. Box 93054 1090BB AMSTERDAM Netherlands www.leaseweb.com descr=LeaseWeb P.O. Box 93054 1090BB AMSTERDAM Netherlands www.leaseweb.com
Here something interesting has happened: since our last transform was ending in a groupby which summarizes the dstip field, ELSA has rolled up all of the whois transform fields and made them an opaque string for presentation after filtering for the non-US addresses. If we want to continue with another transform, it will not do the string summary so that it can pass the results to the next transform in the native format.
So, let's say that we don't want to see the results for 18.104.22.168 because we already know about that one. We can add another subsearch to filter that out
+dstip:22.214.171.124 groupby:site | subsearch(class:url groupby:dstip) | whois | filter(cc,us) | subsearch(class:url -126.96.36.199 groupby:site)
We get back:
Obviously, these are hits from our node's lookups and not relevant. Why did these come up? The subsearch actually ran this query:
class:url -188.8.131.52 groupby:site +(184.108.40.206 220.127.116.11 18.104.22.168 10.56.64.145 22.214.171.124 126.96.36.199)
So, any log in class URL that contains any of these terms (except 188.8.131.52 because it was negated) will match. That means that an URL like
will be found in this match. We don't want that. Luckily, there is a way to further filter this by adding on a field name (dstip) as the second argument to subsearch like this:
+dstip:184.108.40.206 groupby:site | subsearch(class:url groupby:dstip) | whois | filter(cc,us) | subsearch(class:url -220.127.116.11 groupby:site,dstip)
Now we get no results back, which means that there were no hits to any clients visiting sites on the other IP addresses because none of the IP's in the previous search showed up as a dstip in class URL.
Let's look at another example. Find all clients who were attacked by an exploit kit like Blackhole:
+sig_msg:"exploit kit" groupby:srcip
Within those results, find the dstip's that any Java web requests went to:
| subsearch(+user_agent:java groupby:dstip)
Perform a whois transform and only include Russian sites:
| whois | grep(cc,ru)
Now find all sites hosted on these Russian IP's:
| subsearch(class:url groupby:site,dstip)
Found some more bad sites!
Correlation is a powerful tool for combining multiple simple notions to create a very sophisticated question. The ELSA utility transforms, like whois, provide a good way of whittling down large amounts of data into something interesting for use in alerting and reporting. It is sometimes the only way to tease out bits of data that would otherwise blend in with the vast amount of similar logs.