Transforms
Transforms are post-search display filters that data is passed through to modify what is shown the user. For ELSA, this means that when the search engine is finished with its normal search, it will pass the list of results to a chain of filters strung together with pipes which can add fields or filter results. Currently, ELSA ships with the following transforms which add fields:
- Whois
- DNSDB (via account with dnsdb.isc.org)
- CIF (via local instance of Collective-Intelligence-Framework.googlecode.com)
and the following utility transforms:
- grep
- filter
- sum
Below is an example of a simple search which has Whois data added to each entry:
The query is a standard ELSA query, but using the pipe character (“|”) it is possible to send the results through the “whois” transform, which will add fields like country code “cc” and description “descr.” This can be crucial for analysts as they investigate because it saves them having to lookup IP addresses individually. As with everything in ELSA, the goal is to make delivering data as quick and effortless as possible to encourage investigation above and beyond typical searches. This makes for great hunting.
Hunting++
One easy way to start looking for anomalies in your IDS logs is to match up otherwise innocuous or “policy” signatures with anomalous locations or networks. In the screenshot below, you can see a search for executables downloaded from country code “ua” (Ukraine).
Using the info plugin (by clicking the link next to the log entry) shows the StreamDB output
in which you can clearly see that this is a download of “info.exe” caused by a Blackhole Exploit Kit due to the tell-tale “w.php?f=16&e=0” (which indicates that is is an update to an already-infected machine).
Setting Up Transforms
By default, whois will work with the default dummy configuration which declares local RFC1918 space as “MyOrg” as specified by your “known_orgs” entry in the elsa_web.conf under “transforms/whois.” The local IP will have the “customer” field added, which is specified by the “known_subnets” entry. ELSA ships with examples included, so you should only have to edit what already exists.
If you're upgrading ELSA, you may need to add a Perl module for faster result caching, as described below. To add it, run “sudo cpanm CHI::Driver::DBI” or ELSA will use memory-only caching, which will not persist across Apache restarts. This module is included in the install.sh.
Using Transforms
The main caveat to transforms is that you cannot search solely based on them. That's because they are applied after searches are finished, and can only be used to display additional information. This also means that the transforms are only applied to the number of results being returned. You may need to increase the number of results using the “limit” keyword to have the transforms applied to up to 1000 results. As an example, if you wanted to find all downloads with IP addresses related to Google, you would do this:
+sig_msg:exe limit:1000 | whois | grep(descr,google)
The main search is for 1000 alerts with “exe” in the message, which then has the whois fields added (cc, description, org), and then only records with a description matching “google” are returned. The grep and filter transforms are case-insensitive regular expressions on both the field name and the field value. If you want to search all fields for a given value, you can use the dot “.” instead of a field name, like this:
+sig_msg:exe limit:1000 | whois | grep(.,google)
All transform information is cached in the database to speed up result times. Generally, whois lookups are quite fast, but not nearly as fast as ELSA results usually are. However, if you run a search with a transform and then run it again with a different grep, filter, or sum, it should return almost instantly because it is using cached data.
Other Transforms
ELSA also ships with the dnsdb and cif transforms in addition to the whois transform. In order to use the dnsdb transform, you will need a working API key from ISC, which is free to non-profits. Contact them at dnsdb.isc.org for additional details. Below is a screenshot of what it looks like when you add on the passive DNS hosts to your search by passing the results through dnsdb:
It's verbose, but in combination with grep or filter, it can be very helpful.
The cif transform hooks into a local instance of the Collective Intelligence Framework which applies blacklist information as well as whois info to your results. Here's another hunting trip, this time looking for any IP's listed as type “malicious” from which there were downloads:
To use the CIF transform, you'll either need a local instance setup (highly recommended!) or a friendly org that is willing to give you an API key to their instance. In either case, edit the elsa_web.conf file to add the server name, IP, and apikey under “transforms/cif.”
Adding Transforms
Creating your own transforms is not entirely trivial, but isn't too hard. Take a look at the current ones in the web/lib/Transform directory, and with a lot of copy and paste, you should be able to hack your own together. If you get stuck, please drop us a note on the mailing list (http://groups.google.com/group/enterprise-log-search-and-archive).
