Interview with Brian Hatch

LQ) Tell us a little bit about yourself. How did you end up a security guru? Any advice for people who are interested in starting in "the business"?

BH) I was always a paranoid security freak, though I didn't know it until much later. Even when I was 6 or so I had home-made locks on my bedroom door, Tripwire-like devices I could use to see if someone had opened my closet, and other stuff that was very unnecessary for someone with nothing interesting whatsoever. Building better and more foolproof and complicated systems was great fun for me, even if none of it was useful in the least.

Advice? If you want to get into security, you must build an immediate distrust of everything you hear and see. (This also works well when listening to politicians.) When developing anything, be it your security policy or your random email signature generator, you need to take the stance "What could go wrong? What weird situation/input/etc could cause this to fail? Have I set up enough barriers? Have I checked the exit status of each and every command, including 'print/printf'?" Never assume that something you write for a normal user will never be run by root, for example. Never assume something that, today, is only executable by trusted administrators will never be accessible to an attacker. Perhaps those admins become untrustworthy, or their account gets compromised, or you need to allow access by less-competent admins.

To get into the field itself (which seems at the moment to be saturated with true experts at one end and actors at the other, with few people in between) you need to get intimate with code. Go out, grab a piece of software, and bang away at it and see what breaks. A core dump is a bad sign for a programmer, but an indicator of potential exploit for an attacker. Be that attacker on your own systems, see what you can do. Lock your root password on your machine and see how long it takes you to get it back, then close that hole. A month later, lock it again and see what you now have at your disposal.

While there are classes out there in "hacking", be they at conferences or at college/university/etc, many are taught by the same instructor who just learned C programming last year, or the Windows administrator who just grabbed an IIS security book at Amazon. While there are some really top notch instructors out there, the majority of what you'll find is not worth your time, and will give you a false sense of competency.

Download a known-buggy program. Read up on the vulnerability that's been announced, and write your own code to exploit it. Then move onto a new program. Try breaking them with less and less google searches for hints. And don't feel afraid to talk with other folks while you're learning. You'll get a lot of rudeness from some people, so develop a thick skin first.

LQ) Next to your work for Onsight, lecturing, being an active member of the security community and author of "Hacking Linux Exposed", what do you contribute to the community?

BH) Some time in late January or early February, my fiancee and I will be releasing two new projects, currently code named "Baby2" and "Baby3". (Our previous joint project, "Baby1", now known as "Reegen" has been very well received.) I don't know if more of my DNA is a positive contribution or not -- I'll let the community judge that.

As to my Linux/Security contributions, I wrote Hacking Linux Exposed and co-authored Building Linux VPNs for those of you who like dead trees. I also write a bi-weekly email column "Linux Security: Tips, Tricks, and Hackery". I'm also co-maintainer of Stunnel, the "Universal SSL Wrapper" which you can use to protect your cleartext communications without modifying your code itself. Stunnel is used to protect many POP and IMAP servers, for example, and I use it a lot for generic SSL tunneling.

My most recent and visible contribution was with Nmap-3.45. I spent about two weeks badgering Fyodor, trying to convince him that integrating SSL into the version fingerprinting was a good idea. I had him convinced that it was a good idea, but he figured he'd add it into a later release. In the spirit of 'putting my code where my mouth is', I spent a few late nights building in OpenSSL support, which he then massaged into the main code base. Lots of testing, benchmarking ciphers speeds to allow the fastest scanning, supporting session ids cleanly for even faster scanning, etc. When 3.45 came out, I was a happy man. Hopefully other people will benefit too.

LQ) What tools do you consider essential for getting your work done? Any tools you think are under-rated or unknown that you would like to point out?

BH) My work is so diverse that I don't even know how I'd categorise it. Network administrators, be they security or no, need tools like Nmap, Netcat, and Mtr, my favourite traceroute-like program, and a sniffer or two like tcpdump or ethereal. Administrators need local file integrity tools, like AIDE or Tripwire, and good post-compromise tools like chkrootkit or a good bootable CD for intrusion investigation like FIRE or even KNOPPIX in a pinch. And for goodness sake, come up with a good logging policy (both local and remote!) and check your logs! Even fresh administrators can go far using simple perl-based tools like swatch. Security administrators need to scan their own networks for vulnerabilities with tools like Nessus, watch for intrusions with IDSs like Snort. If they really like being paged a lot at night, they certainly need a good port scan detector like scanlogd

All of these tools are nothing new. However it's important to know what they can do well and how to use them to the fullest. For example, it's trivial to script Nmap to scan your network ever night, save the output to a file, and do a diff against last night's run to see if there are any changes. Simple to write, quick to run, and a total cost of $0.

If you want your machines to be secure, even in the face of a root compromise, you want to harden your system. At one end of the spectrum we have helpful automatable tools like Bastille that not only lock things down but teach you a lot along the way. At the other end you have advanced security models that go was beyond POSIX file permissions and capabilities, such as RSBAC, SELinux, Grsecurity and LIDS, and buffer overflow-stoppers like PaX. Some of these require more or even significant configuration before you reap the full benefits of these advanced security patches, but if you take the time you can have a really rock-solid machine.

LQ) As a security expert what is your stance on the ever popular bind vs. djbdns topic? What about postfix/sendmail/qmail? Are there any "standard issue" programs that you think people should stay away from?

BH) BIND: Big huge daemon that does everything. DJBDNS: Small pieces each running chrooted in their own directories doing very specific tasks (authoritative server, caching server, transfer server, logger.)

Sendmail: Big huge daemon that does everything. Postfix and Qmail: Small pieces each running chrooted in their own directories doing very specific tasks (SMTP server, queue manager, local email acceptor, address validator/rewriter, bounce creator, local delivery.)

So, let's see - BIND and Sendmail both have a monolithic approach - at all times, some part of the server is running as root. DJBDNS, Postfix, and Qmail always have each piece doing just a little bit of the job, with only the permissions it absolutely requires. BIND and Sendmail have had numerous security vulnerabilities, including root compromises. DJBDNS, Postfix, and Qmail have had one that I can think of - the possibility that mail might be delivered twice under really unlikely and actively alarm-sounding circumstances.

DJBDNS is, hands-down, the best choice for doing DNS. For mail serving, I use Postfix because I prefer having one config file (which comes from source with excellent comments) and it is available for many Linux distributions in binary form, which means it's easy for me to recommend even for the timid. Qmail, while powerful, elegant, and rock solid, doesn't come pre-compiled with any Linux distributions I can think of due to the license, and is architected in a way that is consistent with other DJB software (including DJBDNS) but is harder for many administrators to wrap their head around at first.

LQ) What do you consider the worst security advice that you see consistently given?

BH) For a while, I would still say "Choose a password that is 6-8 characters long, not based on a dictionary word, blah blah blah ..." without specifying that this only applies to DES hashed passwords. These hashes (traditional 'crypt(3)' form) used the first 8 characters of your password to create a 56 bit key (specifically, it used the bottom 7 bits of the 8 characters) which it used to encrypt a known string (usually all zeroes).

When Linux distributions started supporting MD5-hashed passwords, I still failed to note the difference consistently enough. MD5 is a true hash algorithm, while DES (an encryption algorithm) was used in a way to provide a hash-like algorithm. However MD5 can take an infinite length password, and DES could only take 56 bits. Luckily that's in the past.

Unfortunately, I still do occasionally have my students open up a shell as root on their lab machines, rather than do everything via sudo. True, I tell everyone that this is just to simplify getting their work done in a training environment, however I know it means that they're likely to take that same shortcut home with them when the leave. Sudo offers an audit trail. Logging in as root doesn't, and opens you up to potential local exploits by tricky users, and increases your chances of making errors on your system.

LQ) What do you consider the top 3 must-have apps to help harden the system post-install?

BH) rpm -e / dpkg -P / rm / etc. The less you have on your system, the less can be exploited, and the fewer tools a cracker can use if they do manage to get on your system. Distros still ship with far too many things installed by default. If you plan to use Gnome, do you need KDE? No, but selecting "desktop install" from a pretty menu at install time will probably give you both. Delete everything you can. It also makes backups faster.

For the new administrators, Bastille is a must-have, since it walks you through many pieces of your system to help you harden them.

Some sort of log monitoring software, and file integrity tools, are also requirements. You can't know your machine has been compromised if you're not watching your logs. You can't know what happened after a compromise if you don't know what your system looked like before they got in. I prefer swatch and AIDE respectively, but almost any tool will do.

LQ) Any security-enhancing kernel patches you favor over others (Open Wall, Grsecurity, LIDS, RSBAC, LSM, Medusa, etc)? why?

BH) The Linux kernel patch from Openwall (restricted links in /tmp, restricted /proc, etc) is always a good choice, because it really doesn't interfere with any normal processing, and yet can have a great benefit for any system. On my sensitive systems (and I consider anything that I use to be in that category) I prefer LSM (Linux Security Module) with LIDS (Linux Intrusion Detection System) currently, because I've been using it the longest and have an extensive restrictive ruleset. I've been playing around with Systrace quite a bit recently as well, especially for executing untrusted code in a safe/auditable manner. I really like the idea of more advanced models like SELinux, but the amount of user-space code recompilation is a big drawback in some environments.


LQ) What are the top 3 things to do/look for when you think a box is compromised?

BH) First, start tcpdump on a box next to the compromised box if you can, snaggingeverything. Look at the dump files periodically with ethereal or other tool to see if you can see any suspicious traffic. If the box is owned, you may be able to see how they're controlling your machine. Perhaps it's a simple SSH login, or perhaps its a more obscure covert channel like IP over ICMP. Whatever they use to access your machine may be hidden from the host itself using a rootkit or LKM.

If you want to stop the intrusion immediately, and don't care about finding the attacker, then the best solution is to immediately get on console and reboot off of pristine media so you're sure you don't have any backdoors/rootkits/LKMs/etc colouring your view of the system. However this could very well deprive you of crucial information about what has been going on, and if you wish to find out the source of the attack for legal reasons or just to know what to expect, this is not a good solution. The attacker will almost instantly know they've been caught. If you're just another compromised host, they'll move on. If they're targeting you specifically, they'll know you suspect something and be more stealthy, or take action against other hosts of yours that they control. If they're malicious, they may have actions set to run on shutdown or bootup to wipe your hard drive. So no matter what course of action you take, if your machine is compromised you are at risk of further compromise or data loss.

I prefer to investigate discreetly while the computer may still be under the attacker's control. This is not always the best option, so choose it only if you're willing to accept the risk of retribution should the attacker figure out you're aware of him and decide to delete everything, for example. Did I mention Backups? This is a great time to do a quick rsync of the system from a remote machine, and it may just look like some remote cron job.

One obvious tool is chkrootkit. It can help find rootkits it knows about, as well as determining when other suspicious situation is at hand, such as an LKM that is hiding processes. Any time you find that the kernel itself has been compromised, you know you're not going to be able to definitively determine what's actually occurring, but when faced with just user-land modifications you may be able to determine what's happening on your live system by uploading pristine binaries of ls, lsof, netstat, ps, top, find, and friends and comparing their output to the compromised binaries. Discrepancies will quickly show you where the intruder is storing their files and what processes they're running.

If you discover LKMs or suspect your kernel itself has been compromised, your investigation will only be reliable if you boot off of read-only media, such as a CDROM. Keep a couple around, such as a Knoppix CD, or better yet a System recovery cd or FIRE/etc.

LQ) People consistently point to the fact that you can always audit the source code yourself as a big strength of OSS. If you can't actually read code this becomes a moot point though. In this case what do you recommend? Is trusting the authors all you can do?

BH) I'm sitting on a Debian box right now. On my system are thousands of packages that have been compiled by people who are not me, written by people who are not me. If I took the time to audit all the code that's running to allow me to write my email (mutt) with my editor (vim) in this window (Eterm) on this screen (sawmill + XFree86) on this kernel (Linux 2.4.18) on this machine, it would take me several months. No one expects anyone to do this. And I'm still leaving out the BIOS and CPU schematics, which I don't have.

However there are several things I'm running right now that I have audited. For example I'm very familiar with Stunnel (and maintain the 3.x branch now.) Whenever a new version came out, I could just check the differences between the old and new branches, providing me a small set of changes to vet. Again, not something that everyone can do.

However, I'm not the only one checking out the source. The package maintainers at Debian, Red Hat, SuSE, etc are also checking out new versions. Many distribution maintainers become as intimate with the code as the authors themselves. When a bug is found, which sounds easier to a maintainer - to maintain a patch for the official distribution and apply it each time for their distribution when new versions come out, or to get that bugfix applied to the main tree? I'd sure rather have my fixes get applied upstream so I can be done with them, and this is what happens almost every time. It's easier on the individual Linux distro maintainers, and it ends up benefiting everyone because the change gets pushed back into the source that everyone uses. A win-win all around.

So true, not everyone can read and understand the code that they end up running, and not anyone can read all of the code that they end up running. There's a level of trust, and that's no different than when you run proprietary software. The big difference is the number of individuals who do view that code. Let's take a worst case scenario and say that some proprietary program and the comparable Open Source program have the same number of developers. The big win for the Open Source program is that the developers are not all part of a single organisation. They have no strings attached to viewing the code, no threat of being fired if they discover and announce a vulnerability, and no political/internal agenda that can keep them silent. In the Open Source world, what they find can be shared, and if that finding is a bug, it can be fixed. No corporation can stop them.

LQ) Seems the buzz on the security front is shifting from IDS to IPS. Where do you think security is headed both short and long term? What do you consider the biggest deficiency in the current offerings?

BH) IDS is passive - it will alert you to problems that it sees, and you need to decide what to do about it. In the middle of the night, it may be a while before you get up and can check on the problem, at which point you may have been compromised.

IPS is active, and will stop those attacks that it knows about, so you can sleep a little bit longer.

In other words, IDS doesn't stop the problem at all, and IPS only stops the problems it knows about.

Both of these suck.

The ultimate way to secure the network packets going into and out of your systems is to have an inline filter that knows what is good, and allow only that traffic. Today we don't know the next big Windows bug, but if we train a transparent inline security device, such as a Linux box running Hogwash, to know what the legitimate traffic should look like, then we'll have a good shot at being safe when the next attack is launched.

Unfortunately, doing so is very very hard. It's easier for very specific applications. Say you have an old known-buggy web server, and can't upgrade it for some reason -- it's the only thing that lets you run your old dynamic-content application suite, for example. If you watch legitimate traffic to see what kind of HTTP requests are valid, employing regular expression wildcards to handle the variable portions, you can get a very detailed picture of what is correct. Should someone throw an unknown attack at you, it should not match any of the legitimate traffic, and you will have blocked it. However if you used an IPS, and the IPS developers don't know this attack yet, it wouldn't stop the attack.

If you want to sleep very soundly, having an inline filter that works in this way - 'allow good, disallow everything else' is the best plan. Of course, it'll take several weeks of no sleep to get it just right, so there's the tradeoff.

That said, yes I have IDS on my networks. (Squid, unsurprisingly.) I don't think I'd every bother with IPS - it's too much a half-hearted attempt, since it blocks known bad and allows all else, which is the opposite way to write secure access lists.

LQ) Given the average audience of LQ what do you think is the biggest thing we could do to improve security awareness for the average Linux user?

BH) Does it need to be legal? If not, how about having LQ Nmap each client in the background and showing the results on a subsequent page when they're available. So many people are surprised to see how many services they are running. Format the output to have links to a very minimal description of what these protocols are used for. When 'Nmap -sV' notices webmin, have it say "This is a graphical way to administer your machine. If you don't use it, turn it off. If you use Red Hat, click here for info on using chkconfig. If you use SuSE ... If you don't know, try running these commands to figure out your distribution".

Man, that'd be a lot of work. It could certainly be done without the forced Nmap scan, of course, and might not get the LQ administrators in hot water. ;-)

Interesting stuff.
You should have asked if he as a pro really believes the guys at valve are as incompetent as they look or if he thinks that whole thing was just a publicity stunt to buy some more time.

Originally posted by crashmeister Interesting stuff. You should have asked if he as a pro really believes the guys at valve are as incompetent as they look or if he thinks that whole thing was just a publicity stunt to buy some more time.

Well, I'll do my best...

The guys at Valve claim to have been compromised by a bug in their Windows software. I find that extreemly believable. Does that mean they're incompetant? Or do they just have bad judgement in running such buggy security-lacking software in the first place?

I doubt that it was a publicity stunt (most people try to at least have some positive spin on publicity stunts) but I don't put anything past a corporation...

Another good interview Jeremy.

Welcome to LQ Brian and good luck with Baby1 and Baby2

Originally posted by david_ross Welcome to LQ Brian and good luck with Baby1 and Baby2

I've been pointing people to LQ for a while now (Unspawn introduced me to it) but have never had time to post. Between all the mailing lists and my local LUG msgboard and stuff, I just don't seem to have any more free time.

Thanks for the virtual luck - we'll need it.

Baby1, AKA Reegen, is no longer in utero. You can see her and one of her sisters/brothers in one of my presentations at http://www.ifokr.org/bri/presentatio.../mgp00013.html

As I always say - never pass up a chance to show pictures of your kid(s) during a talk. ;-)

what can i say.. fantastic interview. *takes notes* *runs off to google some things* =D

Thanks for the tips. I am going to look into some of the things you mentioned.

Thank you for your opinion.With incompetent I meant that according to their statement they knew that mailboxes where compromised and still didn't take the server off the net and investigate.

Great interview for someone like myself who barely qualifies as a newbie. As I am basically just learning my first Linux distro now, (and I find even that a challenge) the amount of work required to secure a system seems huge! Brian Hatch really seems to know his stuff.

I have been reading all I can on Linux security for over 6 months (while I work at my job in health care) because I have a chance to join in a new startup that has an advanced security product for Linux systems developed by a friend of mine.

This product is along the lines of SELinux that Brian Hatch mentions in the interview, only it is designed to be user frendly. It can be dropped onto any Linux system and convert it into a trusted operating system (contingent upon proper user authorization), complete with mandatory access controls and fine grain auditing of all users. It loads in minutes, requires no recompiling, utilizes pre loaded templates at max security to start. Allows customization on a per user basis with intuitive, interactive GUI. Protects from internal and external threats whether they be from o/s or application vulnerabilities.
There have been no bugs so far in the initial placements (about 5 months).

From where I sit, it seems that this model of security would be preferable to going through everything that the status quo offers. (i.e.IDS/IPS, scanning, hardening etc.) It is commercial, but the time savings it offers easily justifies its cost.

I invite anyone, including Brian Hatch or Jeremy, who wishes to learn more about this new security product to let me know, and I will post the web site info. I don't like to use these forums for advertising; I come to learn. However, there is a need for this product and I don't feel guilty about "spreading the word".

As a matter of fact I have never seen this site before. I just followed the story link from Netsecurity. It seems very good so far. Cheers.

Since I have now received the welcoming e-mail and read all the rules I will not post the web site info on the forum to avoid selling, or the appearance of selling. If there is any interest though I will put something in the ads forum.

It can be difficult to discuss security coming from the point of view of a new security model or product without coming across as advertising. Sorry if I inadvertantly crossed that line.

Originally posted by lady_k what can i say.. fantastic interview. *takes notes* *runs off to google some things* =D

An interview is only as good as the questions asked, and these were really good questions. When LQ asked if I'd be interested, I said they'd need to come up with good ones - I'm tired of the "What's full disclosure?" type. Well, they certainly managed. Where it normally takes me about 10 minutes to answer interview questions, this one took far far longer, and actually required I use my brain.

I like that.

Originally posted by crashmeister Thank you for your opinion.With incompetent I meant that according to their statement they knew that mailboxes where compromised and still didn't take the server off the net and investigate.

Sometimes it's a good idea to keep a compromised machine up and running, so you can catch the attacker and see exactly what they've done. However if you're not properly monitoring the system, then you've made a critical error. You need to have all your cutoffs ready to go in case the attacker does anything you don't like. You've got to be aware of everything that they're doing.

You've got a lot more leeway if you're running a honeypot - it's not the end of the world if you don't properly monitor a system that doesn't have anything special on it. It does put you in a potentialy bad legal sitation, however, and that hasn't been tested in courts yet, to my knowledge.

But if you know the machine has been compromised, you don't put the monitoring in place to see what they're doing, you're either overworked, plagued by bad judgement, incompetant, or all three.

Originally posted by Praetorpal I have been reading all I can on Linux security for over 6 months (while I work at my job in health care) because I have a chance to join in a new startup that has an advanced security product for Linux systems developed by a friend of mine. This product is along the lines of SELinux that Brian Hatch mentions in the interview, only it is designed to be user frendly. It can be dropped onto any Linux system and convert it into a trusted operating system (contingent upon proper user authorization), complete with mandatory access controls and fine grain auditing of all users. It loads in minutes, requires no recompiling, utilizes pre loaded templates at max security to start. Allows customization on a per user basis with intuitive, interactive GUI. Protects from internal and external threats whether they be from o/s or application vulnerabilities. ...

I'd certainly be interested to hear more, but contact me offline. User friendly and security are hard to find in the same package, unfortunately....

Great interview (though I didn't understand half of it, I'm a newbie ). It really shows the questions are relevant and the answers are well thought-through.

One thing ... Your reply to the final question reminds me so much of the "ShieldsUp!" that Steve Gibson is hosting over at grc.com that I just had to post this -- except, if I understand you correctly, your suggestion targets entire services instead of mere ports.

All you really need to make that tool legal is a disclaimer and a "Go!" button. This way people are actively asking for it and can't say they've been unknowingly scanned. (You might, of course, make that button really, really click-me-now-esque enticing! )

Cheers!
KlaymenDK

Very good interview. It was good to hear a bit about linux security from someone not in the mainstream media. Many of the articles that I have read regarding linux security (or comparing/contrasting Linux-Windows security) seem to be very one sided. Most of the author's out there who claim to be security experts seem to have very little real knowledge of Linux security, or have a background in Windows security and expect their knowledge to apply to Linux boxes as well.
It is also refreshing to hear someone give an accurate acount of how much work it really takes to maintain a secure environment. It is a ongoing process that really should be an everyday focus for every admin out there. I don't know how many admins I have talked to (several that I work with, and many more from other organizations) that think that throwing up a firewall, sticking your web/email servers in a DMZ and keeping up with patches is all that is needed to maintain a secure network.

Anyway, just wanted to say thanks for a good read.

slight