In my last post, I described the importance of comprehensive logging in an enterprise and how you can use the open-source ELSA to get your logs collected and indexed. In this post, I'll describe the various things you can use to generate logs so that you have something to collect.
The Haystack
The classic dilemma with log collection is that the volume of ordinary logs drowns out important ones. ELSA solves this problem by allowing an analyst to cut through a massive amount of irrelevant logs and immediately find the sought-after logs. This allows the organization to enable extremely verbose logging from all of their devices without sacrificing the ability to find important logs. That in turn allows verbose logs to assist in investigations when they normally would have been sacrificed for efficiency. As a secondary benefit, it reduces the amount time spent managing logs because no one is tasked with the difficult choices of which logs should be filtered and which should be kept.
Historically, network devices and UNIX/Linux devices have been the main source of syslogs. Network logs are critical to detecting attacks and conducting incident response. In addition to providing network connection records for both allowed and denied connections, other important logs are sent by network devices. For instance, denial of service attacks can produce logs from firewalls indicating that they have reached their open connection limit. A Cisco FWSM will generate logs like “Connection limit exceeded” and should be alerted on using ELSA. Other logs may not be errors, but are anomalies. Specifically, logs regarding configuration changes are helpful for detecting unauthorized access or improper change management.
ELSA can help zero-in on these kinds of logs by providing the negative operator in a search query. If most logs from a device contain a certain string like “connection,” then the query can be altered with “-connection” to exclude all of those. These searches happen so quickly that you can work through adding a half-dozen negations in a few seconds to uncover a new anomaly. The interesting string representing the anomaly can then be added as an alert for the future. In the screenshot below, you can see a series of queries, each with a decreasing number of results (the number in parenthesis on the tab) with an increasing amount of negation.
Collecting Network Logs
Let's start with an example for configuring a Cisco router to log all network connection records to syslog. There's a great example of setting up logging from both Cisco Catalyst switches and routers, but in a nutshell, it involves a single line added for your log host (for example 192.168.10.10):
logging 192.168.10.10
Almost all network vendors provide a way to export logs as syslog. If possible, use TCP to prevent log loss. ELSA will handle either TCP or UDP logs.
Collecting Linux Server Logs
Setting up logging on UNIX and Linux is generally simple, but there are differences in the logging clients used. Standard Linux boxes up until a few years ago used the venerable syslogd agent to preform logging. To forward all logs to a syslog server of 192.168.10.10, you would add this to /etc/syslog.conf:
*.* @192.168.10.10
Then just restart syslogd:
/etc/init.d/syslogd restart
Different Linux distributions may change the restart command or the location of the syslogd.conf. A similar syntax is used for the newer rsyslog.
For Syslog-NG, adding a remote logging destination is a bit more involved, but is still not overly complicated. A typical syslog-ng.conf file (usually located in /etc/syslog-ng) will have a source entry like this:
source src {
internal();
unix-dgram(“/dev/log”);
}
This is the entry that allows it to listen on the /dev/log socket for incoming local logs. To forward all of these logs to a remote server, we want to add a log destination to our remote server 192.168.10.10.
Destination d_remote { udp(“192.168.10.10”); };
Then, we add a statement that forwards all logs:
log { source(src); destination(d_remote); };
Restart syslog-ng:
/etc/init.d/syslog restart
Now the server should be forwarding all of its local logs to the remote syslog server.
Collecting Windows Server Logs
Windows Server 2008 introduced a new feature in which servers can subscribe to logs from another server. Unfortunately, Microsoft implemented this in a proprietary way which means that it is not syslog compatible. Luckily, there is a good solution to this: Eventlog-to-syslog. Evtsys works on all Windows versions and is available for 32- and 64-bit. Installation could not be simpler: download the executable from the site and run it from a command-line:
evtsys.exe -i -h my.syslog.server.address
Done! There are also a number of enterprise options for configuring backup syslog servers, as well as fine-tuning which events are sent through the registry. See the evtsys documentation for more details.
The great thing about evtsys is that in addition to its very small footprint and ease of install, is that it will, by default, log all eventlog categories, including application-specific categories like SQL server. ELSA has a built-in parser for events forwarded by evtsys and will parse them so that producing reports on event ID and other characteristics are possible.
For ultra-verbose logging, you can enable Windows process accounting which will create a log for every process created. This creates a veritable torrent of logs, but with ELSA's horsepower, it will take them in stride, making them available in case of a breech. It's nearly impossible for an attacker to infiltrate a server and do damage without starting any new processes. Logging active directory account creations alone makes this a worthwhile endeavor.
Evtsys works on Windows desktops just as well as servers. Malware hunting is much easier when you have a log of all the processes created on the machine by the installation of a rootkit.
Collecting Miscellaneous Logs
Applications on servers often generate very helpful, verbose logs which provide a critical view into the business logic of the app. The only way to catch particularly sophisticated attacks are through the monitoring of the business logic because no observable exploits or attacks will be used. Unfortunately, most apps log to flat files instead of the system's built-in logging facility, and forwarding flat files is often more challenging than it should be. However, there are a few tricks for sending flat-file logs from a server and streaming them as syslog which I will detail below.
On Windows, you will need to install yet another third-party program to perform the logging. It's called Epilog and it's available from Intersect Alliance. This small program will run as a Windows service and stream all files that match a pattern in configured directories as syslog.
Linux makes this much easier if you have a recent version of Syslog-NG. Check out this excellent post from Syslog-NG's makers, Balabit, on how to setup Syslog-NG 3.2 for forwarding flat-files. Of particular interest is the ability to use wildcards to specify intermediate directories like this:
/var/log/apache2/*/*.log
which would allow a web server with a lot of virtual hosts on it to easily forward all of their logs.
Creating Your Own Log Generators
Sometimes you will find that there isn't a good existing source for the information that you want to get into ELSA. I wanted an easy, efficient way for recording URL's on a network to ELSA to correlate with IDS alerts. Unfortunately, we didn't use a web proxy, so there was no easy way of logging this. So, I created httpry_logger to address this issue. It will forward all requests with the response code and size as a log like this:
10.124.19.12|66.35.45.157|GET|isc.sans.org|/diary.html?storyid=10501&rss|-|Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.14) Gecko/20110221 Ubuntu/10.10 (maverick) Firefox/3.6.14|,org,sans.org,isc.sans.org|301|260|8583
ELSA parses this output and creates fields like this:
host=10.68.2.28 program=url class=URL srcip=10.124.19.12 dstip=66.35.45.157 status_code=301 content_length=260 country_code=US method=GET site=isc.sans.org uri=/diary.html?storyid=10501&rss referer=- user_agent=Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.14) Gecko/20110221 Ubuntu/10.10 (maverick) Firefox/3.6.14 domains=,org,sans.org,isc.sans.org
Notice how the domains field includes the comma-separated list of possible subdomains for easy searching. So, an ELSA search for “sans.org” will return all results for web requests to sites under the sans.org domain.
Log What You Can
Even if you're unable to get every log source you want to stream logs, don't let that stop you from getting the quick wins under your belt by enabling logging on what you can. Remember, the benefits are linear, so the more you're logging the more benefit you're getting. Ignore perfection and concentrate on progress!
