Sunday, March 11, 2012

Detection is the New Prevention

Like most large orgs, over the past few months, we've been dealing with many phishing campaigns which spread from infected webmail accounts, scrape the contact list, and propagate their phishing payload.  The catch is that when the spam comes from whitelisted orgs, the phishing all gets delivered because the spam filters either completely whitelist or are less guarded against mail coming from the authenticated mail servers of large orgs.  As a result, there is a higher success rate right now because so many legitimate accounts are compromised in so many orgs across the globe right now.

After a few painful incidents in which it took days to clean all of the infected accounts, I'm pleased to say we've honed our incident response plan for this scenario to the point where it is very effective:

Prevention
Our user awareness training teaches that any suspicious emails are to be reported immediately as phishing instead of normal spam.  The recent incidents have kept this in the forefront of our users' minds.  In the future, we may begin using testing services or our own SP Toolkit instance to simulate phishing campaigns.

Detection
Our users became our initial detection mechanisms.  With the spam gateways letting lots of phishing messages through and with phishing subject lines that are too varied to filter manually, the last line of defense and chance for detection is the user.

Containment
We streamlined and formalized our procedures for mass phishing scenarios.  Our staff in key roles knew ahead of time exactly what to do and when.  Specifically:
  1. Lock an account
  2. Force a logout
  3. Purge all email boxes of a specific subject line
  4. Monitor for successful phishes
  5. Block related on IP's and web sites

Monitoring for successful phishes is where ELSA comes in.  With ELSA logging URL's, we were able to take a given phishing email and monitor for any access to the phishing URL's.  StreamDB would then be used to pull out the POST data to show exactly which user submitted credentials to the phishing site so that containment procedures could be followed.

However, ELSA gives us the ability to go deeper than that and use generalizations from the phishing campaigns our users notice to find phishing that has gone unreported.

Fishing for Phishing with ELSA
In our case, a user reported a phishing email which had a URL of:
hxxp://www.pirini-trade .hr/narudzba/use/web/form1.html
My staff searched ELSA for:
site:www.pirini-trade.hr
No hits, so that particular email failed in its phishing attempt.  However, what if other phishing emails went out with a similar format?  Let's try some other searches:
uri:narudzba
Nope.
uri:"use/web/form1.html"
Nope.
How about this?
+method:POST +uri:use +uri:form1.html
Nope.
Just this:
+uri:use +uri:form1.html
Ok, now we've got some very interesting hits, but no POST's:
GET www.charlesgorley.ca /form/use/Text/form1.html
That looks very similar, but did anyone post data?
+method:POST +site:www.charlesgorley.ca
Hit!
POST www.charlesgorley.ca /form/use/Text/process.php
So, we can use this for our query:
+method:POST +uri:use +referer:form1.html
Set it to alert, and now we've got some decent coverage for variants of this campaign.  If it continues, writing this into a Suricata alert may even be a good idea.  In the meantime, we initiate our incident response plan for this user account.

Detection as Prevention
So, this is great, but how is this prevention as the account was already compromised?  In the case of this example, the POST we found was two days ago.  The key concept to understand is that even though the credentials are compromised, they are not used immediately.  In fact, we have noticed an average turnaround of six days between account compromise and account misuse.  The criminals harvesting the credentials have staff, code, and infrastructure to maintain just like the good guys, so all but the very best lack automation when it comes to actually utilizing the accounts for further phishing.  In fact, our experience has shown an organizational structure much more akin to sweatshop labor than elaborate and sophisticated technology.  Specifically, it is more economical for the criminals to pay low-wage workers to perform the mundane tasks of exploiting compromised accounts to send spam than it is to write scripts to do it.

How do we know this?  The attackers in our previous phishing attacks have all been from a few distinct geographic locations (Spain, France, and Nigeria) and have a distinct operational signature.  They use a browser plugin called "Crazy Browser" which allows a basic Internet Explorer instance to have multiple sub-windows open at the same time in one IE window.  The most likely reason to do this is for easy copy-and-paste between their personal email from which they receive orders from their criminal boss and the various webmail accounts they're logged into for the purposes of spamming.  The point is, if you need a lot of browser windows open, then you're obviously not doing a lot of scripting, and there is most certainly a human involved in the account use.

So, why is this important in shaping our IR plan?  It means that if we know it will be several days between account compromise and account misuse, we can lock down the account before it is misused with sufficiently fast detection.  Yes, the initial prevention via spam filter failed, and the phishtank.com coverage did not show up until a day after the phish was delivered, but from a business perspective, we were able to detect the compromise in time to prevent it from becoming a problem for the business.  ELSA enhances our detection abilities by letting us retroactively investigate phishes based on new leads.

I'm not the first to suggest that detection is the new prevention, but I think it's important to use examples to show how the two are intertwined, and what that means for the business.  Specifically, it means that visibility and analytics are more valuable than (or at least necessary alongside) traditional preventative measures, and we as security professionals need to arm our orgs with the tactical resources to provide these capabilities.

Thursday, March 8, 2012

Correlation in ELSA

In my previous post, I showed how complex the data sets in security have become.  In order to cope with this amount of heterogeneous records, powerful correlation capabilities are required.  I'm pleased to announce that as of today, basic correlation is now possible in ELSA through the new subsearch() transform.

Let's look at an example.  A recent investigation was focusing on what sites were hosting exploit kit materials.  We had a starting point in that we knew a given IP address was hosting an exploit kit, and we wanted to see what other sites hosted related material.  The initial search was:

+dstip:82.192.87.28 groupby:site
   
freshnewstoday.org
893g.in
panam-airline.ru
www.fruitcoctails.com
hife.in
hub7.yourgreatfind.info

So, the question was, what other IP's were also hosting these site names?  This will lead us to other hostile IP's which could be actionable intelligence for blocking at the firewall.  For this, we use the new subsearch() transform:

+dstip:82.192.87.28 groupby:site | subsearch(class:url groupby:dstip)

Which yields all of the unique IP's:

82.192.87.28
218.85.136.44
199.19.53.1
208.67.222.222
8.28.16.201
123.108.111.67
199.254.31.1
74.54.56.23
199.249.112.1
199.19.54.1
184.173.149.222
192.33.14.30
216.69.185.8
8.28.16.203
125.19.40.90

Now, let's say we are only concerned with the non-US IP addresses.  We can apply one of the previously existing transforms to do a whois lookup followed by a filter:

+dstip:82.192.87.28 groupby:site | subsearch(class:url groupby:dstip) | whois | filter(cc,us)

123.108.111.67 org=PANGNET cc=HK name=Pang International Limited descr=Pang International Limited
125.19.40.90 org=EXL-9803-nod cc=IN name=EXL EXL, A-48 SECTOR-58 NOIDA Uttar Pradesh India Contact Person: Mr.Neeraj Email: Neeraj.Jain@exlservice.com Phone:981809784 descr=EXL EXL, A-48 SECTOR-58 NOIDA Uttar Pradesh India Contact Person: Mr.Neeraj Email: Neeraj.Jain@exlservice.com Phone:981809784
218.85.136.44 org=CHINANET-FJ cc=CN name=CHINANET Fujian province network Data Communication Division China Telecom descr=CHINANET Fujian province network Data Communication Division China Telecom
82.192.87.28 org=LEASEWEB cc=NL name=LeaseWeb P.O. Box 93054 1090BB AMSTERDAM Netherlands www.leaseweb.com descr=LeaseWeb P.O. Box 93054 1090BB AMSTERDAM Netherlands www.leaseweb.com

Here something interesting has happened: since our last transform was ending in a groupby which summarizes the dstip field, ELSA has rolled up all of the whois transform fields and made them an opaque string for presentation after filtering for the non-US addresses.  If we want to continue with another transform, it will not do the string summary so that it can pass the results to the next transform in the native format.

So, let's say that we don't want to see the results for 82.192.87.28 because we already know about that one.  We can add another subsearch to filter that out

+dstip:82.192.87.28 groupby:site | subsearch(class:url groupby:dstip) | whois | filter(cc,us) | subsearch(class:url -82.192.87.28 groupby:site)

We get back:

apps.db.ripe.net
whois.arin.net

Obviously, these are hits from our node's lookups and not relevant.  Why did these come up?  The subsearch actually ran this query:

class:url -82.192.87.28 groupby:site +(82.192.87.28 218.85.136.44 193.0.6.142 10.56.64.145 125.19.40.90 123.108.111.67)

So, any log in class URL that contains any of these terms (except 82.192.87.28 because it was negated) will match.  That means that an URL like

GET whois.arin.net/rest/ip/218.85.136.44

will be found in this match.  We don't want that.  Luckily, there is a way to further filter this by adding on a field name (dstip) as the second argument to subsearch like this:

+dstip:82.192.87.28 groupby:site | subsearch(class:url groupby:dstip) | whois | filter(cc,us) | subsearch(class:url -82.192.87.28 groupby:site,dstip)

Now we get no results back, which means that there were no hits to any clients visiting sites on the other IP addresses because none of the IP's in the previous search showed up as a dstip in class URL.

Let's look at another example.  Find all clients who were attacked by an exploit kit like Blackhole:
+sig_msg:"exploit kit" groupby:srcip

Within those results, find the dstip's that any Java web requests went to:
| subsearch(+user_agent:java groupby:dstip)

Perform a whois transform and only include Russian sites:
| whois | grep(cc,ru)

 Now find all sites hosted on these Russian IP's:
| subsearch(class:url groupby:site,dstip)

Found some more bad sites!

www.spbtraveler.com
cat.ms.softspb.com
svyaz62.findhere.org
svyaz61.findhere.org
vlhrusbi.athersite.com
sjpvmakd.3d-game.com
www.spbtraveler1.com
www.spbtraveler2.com
91.196.216.152
www.spbtraveler3.com
svyaz54.findhere.org

Correlation is a powerful tool for combining multiple simple notions to create a very sophisticated question.  The ELSA utility transforms, like whois, provide a good way of whittling down large amounts of data into something interesting for use in alerting and reporting.  It is sometimes the only way to tease out bits of data that would otherwise blend in with the vast amount of similar logs.

Friday, March 2, 2012

Picking Up the Pieces

(When AV Fails)

Managing alerts with ELSA makes it easy to unobtrusively monitor for events that aren't necessarily noteworthy in and of themselves.  One such alert I have is for Symantec Anti-virus alerts, which get logged to the Windows eventlog as event ID 51.  Using eventlog-to-syslog, all of our Windows event logs are forwarded to ELSA.  We regularly see cases in which Symantec logs an alert to the local Windows event log but fails to forward the event to the Symantec management server.  This lets us see the alerts that fall through the cracks.  So, using the search:

eventid:51 program:symantec_antivirus

I get an event back that looks like this:

51: Security Risk Found!Trojan Horse in File: E:\Profile\<user name redacted>\Application Data\BE6CCC87FECC\prf275.tmp by: Auto-Protect scan. Action: Quarantine succeeded : Access denied. Action Description: The file was quarantined successfully.

This log was sent by a network file server, not the client desktop.  By the looks of things, AV did its job and this host is fine.  Just to be sure, I wanted a closer look.  Taking the name of the user from the profile directory, I did a follow-up ELSA search for just the user's name:

<user name redacted>

Which returned the domain controller log showing me which workstation they were logged in from at the time:

672: NT AUTHORITY\SYSTEM: Authentication Ticket Request: User Name: <user name redacted> Supplied Realm Name: <redacted> User ID: %{S-1-5-21-3772051035-1815270206-2219246618-5753} Service Name: krbtgt Service ID: %{S-1-5-21-3772051035-1815270206-2219246618-502} Ticket Options: 0x40810010 Result Code: - Ticket Encryption Type: 0x17 Pre-Authentication Type: 2 Client Address: 10.102.210.89 Certificate Issuer Name: Certificate Serial Number: Certificate Thumbprint:

This told me the user was at IP 10.102.210.89.  I wanted to see what IDS alerts we got during that time, so I searched for:

10.102.210.89 class:snort

and got:

[1:2014170:2] ET POLICY HTTP Request to .su TLD (Soviet Union) Often Malware Related [Classification: A Network Trojan was detected] [Priority: 1] {TCP} 10.102.210.89:2676 -> 78.159.112.131:80

Which is suspicious.  However, we've seen plenty of legitimate sites using .su domains, so this in and of itself doesn't prove anything.  Again, we need to investigate further:

10.102.210.89 class:url

shows:

10.102.210.89|78.159.112.131|GET|imagedumper.su|/imagedump/image.php?size=0&imageid=0&resize=0&data=&thumbnail=1|-|B7B2BAC0C0C0A4ABA3C2C09F92908A9185A8C6C6C1C4C7DEC5C7C3DEAE8FC2C6C0C4\\0\\AK-81|,su,imagedumper.su|200|22528|6869

This looks even more suspicious, but still isn't enough to confirm a compromise.  What else was the host up to?  I wanted to see a list of every site they went to within a minute or so:

10.102.210.89 class:url groupby:site

This returned a list of about a hundred sites, including some very suspicious looking site names:

75686f68.kz
626f6f686f6f.ws

Something is definitely up.  So, what was the content returned by the request to imagedumper.su?  I use the StreamDB plugin to quickly pull the packet content, which shows that this was actually an executable download:

Returning 4 of 2 at offset 0 from Fri Mar  2 07:51:51 2012 to Fri Mar  2 07:51:51 2012 (92 ms)

2012-03-02 07:51:51 10.102.210.89:3812 <- 78.159.112.131:80 5s 22968 bytes Time cutoff

GET imagedumper.su/imagedump/image.php?size=0&imageid=0&resize=0&data=&thumbnail=1

oid=18043-3966842627-452-0

GET /imagedump/image.php?size=0&imageid=0&resize=0&data=&thumbnail=1
Cache-Control: no-cache
Host: imagedumper.su
User-Agent: B7B2BAC0C0C0A4ABA3C2C09F92908A9185A8C6C6C1C4C7DEC5C7C3DEAE8FC2C6C0C4|0|AK-81
X-HTTP-Version: 1.1



200 MS-DOS executable PE  for MS Windows (GUI) Intel 80386 32-bit, UPX compressed

oid=18043-3966843079-22968-0

200 OK
Date: Fri, 02 Mar 2012 16:04:33 GMT
Retry-After: 61867
Server: Apache/2.2.16 (Debian)
Content-Length: 22528
Content-Type: text/html
Content-Disposition: inline
Content-Transfer-Encoding: binary
X-HTTP-Version: 1.1
X-Powered-By: PHP/5.3.3-7+squeeze7

MZ......................@.............................................    .!..L.!This program cannot be run in DOS mode.

$...........A...A...A...8...@.......@...RichA...........PE..L......C.....

So, I extract the executable using the object ID: streamdb/?oid=18043-3966843079-22968-0 and save that to my analysis machine.  I then upload it to the malwr.com online sandbox:

http://malwr.com/analysis/00b714468f1bc2254559dd8fd84186f1/

This shows me that the file should make a DNS request to 1051811156619.com.  Did this happen?  A quick ELSA search through Bro DNS logs confirms that it did:

1051811156619.com

shows:

1330696345.046917|2NMPpT5BTo|<DNS server redacted>|28707|209.236.76.107|53|udp|52601|1051811156619.com|1|C_INTERNET|1|A|0|NOERROR|F|T|F|F|F|0|207.112.46.80|3600.000000

So, our host did in fact execute the binary and make the DNS lookup, despite the Symantec log indicating that it was quarantined!  What did our host do with the answer it got?  Another ELSA search for the DNS answer IP:

207.112.46.80

shows:

Deny tcp src inside:10.102.210.89/3820 dst OUTSIDE:207.112.46.80/20001 by access-group "CSM_FW_ACL_inside" [0xfeb273c6, 0x0]

So, our strict outbound firewall policy is denying our host from connecting to this IP on a weird port.  The host is definitely compromised, but the command-and-control (C2) channel is being blocked.  Based on this, I decide that we need to have an IDS signature for this.  The question is, was the imagedumper.su web request a check-in or was it just a loader?  A quick ELSA query

+uri:"/imagedump/image.php"

shows that there haven't been any other requests to any other sites using that URI and the host makes this request regularly, so this is a good one for a sig.  As a result, I submitted the following to Emerging Threats:

alert http $HOME_NET any -> $EXTERNAL_NET any (msg:"ET TROJAN Win32/Kryptik.ABUD Checkin"; flow:established,to_server; content:"/imagedump/image.php"; http_uri; reference:url,http://malwr.com/analysis/00b714468f1bc2254559dd8fd84186f1/; classtype:trojan-activity; sid:x; rev:1;)

So, what appeared to be a simple case of AV quarantining a virus proved to be an indicator of a full compromise which has been going on for days.  Performing this kind of investigation without ELSA and StreamDB is certainly possible, but I guarantee it would've taken longer than the five minutes it took me, beginning to end.  In total, the following information was needed:
  • AV console logs/alerts from file server
  • Windows logs from AD controller
  • IDS logs
  • URL logs
  • DNS logs
  • Full packet capture
If any one of these information sources had been missing, then we would not be able to confirm a compromise.  This just goes to show how complicated the security landscape is getting, and how much effort it takes to get it right.

Wednesday, February 29, 2012

Online Malware Analysis Sandbox Comparison

At the end of January, I decided that I wanted to see if malwr.com, which runs the Cuckoo Sandbox, was as good as other online sandboxes.  This would be important information because the Cuckoo Sandbox can be run locally on your own machines as your own personal sandnet.  If it was proven to be as good as the other online sandboxes, then it would be worth setting up our own Cuckoo farm.

First off, this was an unscientific study done by my analysts with some submissions from other participants on the Emerging Threats mailing list.  With less than one hundred samples analyzed, it's hardly a significant sample size, but it was enough to give us a good representation of what we should expect for our use case.  The files we submitted were collected from drive-by-download exploit kits in the wild.  See the raw data for Trojan family names, MD5's, and other specific information.  Our scoring was as follows:

-1: Technical failure (web site down, no results of any kind returned, etc.)
0: Error when running
1: Results returned, but no network info
2: Results with network info (DNS, HTTP requests)


What we found was that Malwr.com was essentially just as good as Anubis, which means that it's worth running our own local Cuckoo Sandbox for automated analysis which could be fed by Suricata's new file-extract feature, Bro's file-extract feature, or StreamDB object extractions.

Saturday, February 25, 2012

New ELSA Log Parsers

ELSA is growing!

By popular demand, I've added a number of new parsers to the ELSA repertoire to support parsing fields from the following devices:
 - Fortinet (URL, traffic)
 - Checkpoint
 - Palo Alto (URL, traffic)
 - Barracuda (scan, receive, send)
 - OSSEC Windows logs (automatically appears as class Windows)

Parsed logs will have fields available for ad-hoc reporting and transforms.  For instance, now that srcip and dstip are available, the whois or cif transform can be used with these logs.

In order to use these classes, you will need to uncomment the class definitions in the node/conf/schema.sql file and manually do an INSERT of those classes.  This is to prevent those of us who do not have these devices from having a cluttered menu.  In the future, I'm hoping to have a much nicer way of administrating which classes are visible.

I've also added an updated syslog-ng.conf file which should handle Cisco device logs much better with the varied types of timestamps they send.

Lastly, I've updated the ELSA online documentation to show how to create your own parsers.

As always, if you have any questions or problems with ELSA, please let us know on the ELSA mailing list.

Sunday, January 29, 2012

Hunting with ELSA Transforms

One of the talks at the Bro Workshop 2011 showed all of the amazing things you can do with Bro logs when manipulating them in Splunk by changing the content of the fields displayed. This reminded me that I've been planning on adding transforms to ELSA for some time, and I've finally got the first working version available!

Transforms

Transforms are post-search display filters that data is passed through to modify what is shown the user. For ELSA, this means that when the search engine is finished with its normal search, it will pass the list of results to a chain of filters strung together with pipes which can add fields or filter results. Currently, ELSA ships with the following transforms which add fields:
  • Whois
  • DNSDB (via account with dnsdb.isc.org)
  • CIF (via local instance of Collective-Intelligence-Framework.googlecode.com)
and the following utility transforms:
  • grep
  • filter
  • sum
Below is an example of a simple search which has Whois data added to each entry:

The query is a standard ELSA query, but using the pipe character (“|”) it is possible to send the results through the “whois” transform, which will add fields like country code “cc” and description “descr.” This can be crucial for analysts as they investigate because it saves them having to lookup IP addresses individually. As with everything in ELSA, the goal is to make delivering data as quick and effortless as possible to encourage investigation above and beyond typical searches. This makes for great hunting.

Hunting++

One easy way to start looking for anomalies in your IDS logs is to match up otherwise innocuous or “policy” signatures with anomalous locations or networks. In the screenshot below, you can see a search for executables downloaded from country code “ua” (Ukraine).
Using the info plugin (by clicking the link next to the log entry) shows the StreamDB output



in which you can clearly see that this is a download of “info.exe” caused by a Blackhole Exploit Kit due to the tell-tale “w.php?f=16&e=0” (which indicates that is is an update to an already-infected machine).

Setting Up Transforms

By default, whois will work with the default dummy configuration which declares local RFC1918 space as “MyOrg” as specified by your “known_orgs” entry in the elsa_web.conf under “transforms/whois.” The local IP will have the “customer” field added, which is specified by the “known_subnets” entry. ELSA ships with examples included, so you should only have to edit what already exists.

If you're upgrading ELSA, you may need to add a Perl module for faster result caching, as described below. To add it, run “sudo cpanm CHI::Driver::DBI” or ELSA will use memory-only caching, which will not persist across Apache restarts. This module is included in the install.sh.

Using Transforms

The main caveat to transforms is that you cannot search solely based on them. That's because they are applied after searches are finished, and can only be used to display additional information. This also means that the transforms are only applied to the number of results being returned. You may need to increase the number of results using the “limit” keyword to have the transforms applied to up to 1000 results. As an example, if you wanted to find all downloads with IP addresses related to Google, you would do this:

+sig_msg:exe limit:1000 | whois | grep(descr,google)

The main search is for 1000 alerts with “exe” in the message, which then has the whois fields added (cc, description, org), and then only records with a description matching “google” are returned. The grep and filter transforms are case-insensitive regular expressions on both the field name and the field value. If you want to search all fields for a given value, you can use the dot “.” instead of a field name, like this:

+sig_msg:exe limit:1000 | whois | grep(.,google)

All transform information is cached in the database to speed up result times. Generally, whois lookups are quite fast, but not nearly as fast as ELSA results usually are. However, if you run a search with a transform and then run it again with a different grep, filter, or sum, it should return almost instantly because it is using cached data.

Other Transforms

ELSA also ships with the dnsdb and cif transforms in addition to the whois transform. In order to use the dnsdb transform, you will need a working API key from ISC, which is free to non-profits. Contact them at dnsdb.isc.org for additional details. Below is a screenshot of what it looks like when you add on the passive DNS hosts to your search by passing the results through dnsdb:


It's verbose, but in combination with grep or filter, it can be very helpful.

The cif transform hooks into a local instance of the Collective Intelligence Framework which applies blacklist information as well as whois info to your results. Here's another hunting trip, this time looking for any IP's listed as type “malicious” from which there were downloads:
To use the CIF transform, you'll either need a local instance setup (highly recommended!) or a friendly org that is willing to give you an API key to their instance. In either case, edit the elsa_web.conf file to add the server name, IP, and apikey under “transforms/cif.”

Adding Transforms

Creating your own transforms is not entirely trivial, but isn't too hard. Take a look at the current ones in the web/lib/Transform directory, and with a lot of copy and paste, you should be able to hack your own together. If you get stuck, please drop us a note on the mailing list (http://groups.google.com/group/enterprise-log-search-and-archive).

Tuesday, November 22, 2011

ELSA Beta Available


After two months of solid coding, I'm proud to announce that Enterprise Log Search and Archive (ELSA) is available in beta quality and ships with an auto-installer for many Linux systems. The backend and query code has been rewritten from the ground up with many enhancements. Most importantly, query speed is an order of magnitude faster in many cases, and you can now report on string fields like any other field.  The query interface has been streamlined, along with other interface changes.


The rewrite focused on making the platform more robust by removing many moving parts and streamlining a lot of background processes. The result is agentless, fully independent log nodes which do not need to communicate with any other nodes. This allows for easy distribution of nodes in customer sites. In fact, the nodes can be accessed from as many different web frontends as you like, as they are nothing more than a MySQL query interface to external clients, so customers can have their own interface if they prefer. The web server now handles asynchronously parallelizing queries and aggregating the results, which has improved performance and reliability.

I've also added parsers and classes for the major Bro IDS logs, so ELSA is now a viable frontend for Bro. Below is a screenshot of a source IP address with a report by DNS hostname lookup. This search covered several billion logs and returned in less than a second.

There are many more features, and I will refer interested readers to the new documentation available. I'm very interested in any issues encountered during install and operation, so please let me know if you run into any issues. You may also file a bug report on the project page if you run into any problems.