Help analyzing hackers transcripts

Originally Posted by OlRoy Yeah, I know of another security forum where only approved members can assist other members in recovering from incidents (mostly spyware). It makes sense, and helps prevent a bad situation from becoming worse.

I agree it does make sense and I'd be infavour of that but essentially it goes against the grain with respect to the openness of LQ: we can't elevate a select group of people and deny others to chip in. What if a handler misses something? On the other hand the lack of something like that means that those that do must watch threads to make sure they get there first.

Originally Posted by OlRoy If you guys get enough of these incident threads, perhaps LQ needs an official Incident Report Form.

Thanks, that's a good suggestion. If you look back the past years you'll see the amount of root account requiring breaches of security went down considerably while the amount of daemon-only, PHP-related incidents went up at the same time. As far as I have any overview right now I'd say we're down to less than six major incidents a year in this forum.

Originally Posted by slimm609 Most of the stuff above will keep out the vast majority of script kiddies. If there is someone out there that wants in your system and he is good enough then he will get in. BUT most of the people that have the knowledge to get into a hardened server are not going to bother with it. They have more important things to do then hack _your_ system.

Oh, this isn't true. Not at all.

If it was true, then the only choice would be to turn off our computers; the data can't be protected.

If the system is fully hardened and security is up to date, no one can get in unless they have physical access to the box (if they do, it can't be secured, period), or unless they discover a previously unknown flaw, on the fly.

It is true. There are tons and tons of unreleased 0-day exploits. There is no system that is 100% secure. The only way to be 100% secure is to turn the machine off and lock it in a safe and put the safe into 200 feet of solid steel and to be truthful thats not even 100% secure. no matter how much hardening you do there will always be exploits. End of story. The "true" hackers (that find and write the vulnerabilites) would be able to get in with enough time. The average time between the discovery of a new vulnerability and the public release is 30-60 days

Originally Posted by slimm609 There is alot of time spent into incident response and handling. It is not something that can be learned overnight.

Actually anyone can learn the basics, and not overnight either: most of it requires "just" a structured approach, the discipline to follow it and "the right mindset" (aka common sense). Five minutes tops. Here it goes (note "you" here is the "generic you", OK):

The right mindset starts by seeing who and what is involved. For the "victim" these are bad times: overloaded with work, deadlines on the horizon, harassed by management, users and remote admins, no clear view of what's going on or how to solve it. Help is needed. Fast. For LQ having the right mindset involves more than just roughly "getting things fixed": not only does each incident warrant your full attention because we value each and every LQ member alike but also your posts now will help people later if they find it through say searchengine results. And as you might have seen me posting it in my EOL notices: running GNU/Linux is all about performance, protecting assets and providing services in a continuous, stable and secure way. So for me personally every single root compromise is one too much just because it reflects badly on GNU/Linux regardless how it was done and regardless how badly the admin managed his or her machines.

Often the victim will be stressed out, in a hurry to "do things" (or worse: has various done things already) and so needs steps that have to be helpful, ordered, clear. Because of the importance and urgency of the situation, and more than in threads of any other nature where you can post and walk away, you must be willing to commit to seeing this through and "guide" the victim. If you don't then all sorts of advice will be fired at the victim leaving him/her confused and dragging the case on and on w/o conclusion, without closure. That's what requires discipline. Committing also means "owning" the case. That could involve telling the victim to read required material, revisit earlier posts, speed up posting "evidence" but also point out any omissions or mistakes if anyone makes them.

Do realise that you only have one shot so you have to get it "right". That's also where the common sense part begins and where the first mistake is made. The most common first mistake people make is to give "advice", a "solution". As any good troubleshooter, anyone with even the remotest analytical skills knows, you can't give a solution if you don't have a complete overview of the situation. Most of the time you'll have to ask or press the victim to post nfo.
And any "advice" that involves performing ops on the system could muddle things further and obstruct later analysis (the investigators footprints across the bloodsoaked carpet). That is not always the case but any decision to post steps should take things like that into account. The common second mistake people make is to take things at face value instead of making certain something is what it *says* it is (run for example 'apropos doexec'). Do not draw conclusions too fast, to easily: base your conclusions on facts, not fiction.

When performing incident handling myself I always try to adhere to a structured approach (as posted in earlier threads). In talking to my fellow moderators and you (in general) I've always promoted, asked people to post the most important URI's (Intruder Detection Checklist (CERT): http://www.cert.org/tech_tips/intrud...checklist.html and Steps for Recovering from a UNIX or NT System Compromise (CERT): http://www.cert.org/tech_tips/root_compromise.html) even if they do not have the knowledge or time to "own" the incident. The reason for posting is the victim gets focussed on doing things "the right way", steps to perform should result in "evidence" to asses and the list keeps things flowing. The structure is nothing special, just there to help you make certain you understand the situation and know the goal. Then determining the path between those two is subject to the situation.


There may be exceptions, like incidents that are so obvious the "don't give advice" part doesn't apply, but that is not the argument. And while the technical analysis part requires more in-depth knowledge, starting incident handling "the right way" doesn't. It just requires you to be there and lead things on. In fact you could be an incident handler and call in experts where necessary (usually unnecessary as they'll be circling threads like those anyway). This is the mindset I've tried to do incident handling with as long as I've been a part of LQ with the goal of taking things to a qualitative better level than other fora like LQ. Those who think that this is convoluted, contrived, unprofessional, over the top fascist or too much hassle just don't get the point.

Originally Posted by jiml8 Oh, this isn't true. Not at all. If it was true, then the only choice would be to turn off our computers; the data can't be protected. If the system is fully hardened and security is up to date, no one can get in unless they have physical access to the box

False.

Why do projects release security patches? Because someone found a flaw in a system that was already up to date and totally impenetrable (by your reckoning). There would never be any patches if simply having the latest release made you invulnerable.

The fact is that no matter how thorough you think you've been, there's a flaw somewhere, it's just that no one has found it and/or made it public yet. In fact, you said it yourself:

unless they discover a previously unknown flaw

but this doesn't mean that a particular attacker would have to accidentally stumble upon a flaw at the very moment they're looking at your machine. They may very well have known about the particular flaw for a long time and they are keeping it to themselves (perhaps with a small group of friends) in order to profit from it. It's in their best interest to not report it to the proper maintainers.

Just because a flaw exists doesn't mean anyone knows about it; just because someone knows about it doesn't mean they will report it; and just because a flaw is being exploited doesn't mean anyone has noticed it.

So accept the fact that if your box is connected to a network, it is probably vulnerable to compromise. The goal is to deal with all the known vulnerabilities, put strong mitigation steps in place against unknown vulnerabilities, and be ready to recover your system if all else fails.

Originally Posted by unSpawn Actually anyone can learn the basics, and not overnight either: most of it requires "just" a structured approach, the discipline to follow it and "the right mindset" (aka common sense). Five minutes tops. Here it goes (note "you" here is the "generic you", OK):

Your right you can learn the basics in 5 minutes and type some commands that people are telling you to run. Most 3 yr olds could. But to actually understand the structed approach you have to have a good knowledge of the kernel and systems. You cant just read a document then say i can recover from attacks correctly. There is lots of trial and error and understanding OS's from a security standpoint. The ability to understand the OS's from a security aspect it a small group.(Not attacking or bashing anyone here). I am just saying that the ability to learn is not there but the time and devotion is a thing that alot of people dont want to do. The proof to that is that fact that the Security Engineer positions are very well paying and they are alot less security positions then there are administrator positions. They are 2 different mind sets.


What do you do if the CERT checklists dont work or don't give the results your looking for. Does that mean that you haven't been hacked or compromised?

This is where the knowledge that can't be read overnight comes in.

When getting hacked I usually poke around to try and get the best idea of what is going on, then format and reinstall. It's better off not to take any chances. They could hide a script somewhere with 1000 linux stock scripts but edit one of those and make it execute at a certain date only, there's so many things they could do to put their hack "to sleep" so it reactivates later on.

Good example is a while back my site got hacked by some taliban hacker (some kind of anti US protest - I'm Canadian, why did he attack me? lol). The hack was actually not too damaging at first, but then as I looked closer I found a php based trojan. Basically a script thrown in my forum with a name that looks legit, but if you execute it, there was all sorts of fun little toys like a mysql dump/query form, file manager etc... So even after reuploading the damaged part of my site they could of still gotten in. This was a shard hosting package and the hack was probably done due to a chmodd 777 file (non suphp environment) so my version of format was to delete my entire home folder and re-upload.

Backups are not only important because of disk failure, but because of hackers too. Glad I do mine.

Originally Posted by Red Squirrel Basically a script thrown in my forum with a name that looks legit, but if you execute it, there was all sorts of fun little toys like a mysql dump/query form, file manager etc...

You mean a PHP shell?

Originally Posted by Red Squirrel Backups are not only important because of disk failure, but because of hackers too. Glad I do mine.

Good you do, yes. But how do you verify that what you restore is only what you want to restore?..

Originally Posted by unSpawn You mean a PHP shell? Good you do, yes. But how do you verify that what you restore is only what you want to restore?..

What I restore comes from my home PC which is on a secured LAN. In fact if I want to be very paranoid I can just grab one of the external media backups of my home network which would also have the backups of the sites.

In the event of a hack you cannot trust any backups that were done on that same server or any server it clearly talks to (so say you have a rsync script going to another server, they hacker can find that script and just hack that server too). Best to even delete them once everything is restored to ensure they are never accessed by error.

Yes, I understand that. But what I mean is how do you verify (check, validate, make certain) that the files you restore are just those and nothing more? I mean that cvs_null.jpg might by filename easily slip through while it is one of those PHP shells.

slimm609

It is true. There are tons and tons of unreleased 0-day exploits.

This I doubt. Well, with Windows it is an imponderable, but with Linux or BSD, I have to take the position that many eyes make for shallow bugs.

chort

Why do projects release security patches? Because someone found a flaw in a system that was already up to date and totally impenetrable (by your reckoning).

Sure, there are patches. But it takes the uber-black hat to have an unknown exploit that he can use before it is discovered and patched. Not to say this is impossible, but the probability is low and the likelihood of it happening to one of your or one of my servers is therefore small.

Further, the probability that such a defect is even exploitable from outside is greatly reduced if the server sits behind an NAT router.

Finally, new-generation processors are providing protection for data areas of memory, forbidding code execution in those areas, which tends to make exploiting buffer overflows next to impossible.

The result is that a properly secured server, where the security patches have been maintained, is perhaps not impossible to crack from outside, but extremely unlikely (probability approaching zero) to be cracked.

Originally Posted by jiml8 slimm609 This I doubt. Well, with Windows it is an imponderable, but with Linux or BSD, I have to take the position that many eyes make for shallow bugs.

Or:

There's so many people who will be checking this, why
should I bother. Now if "everyone" thinks like that the
many eyes have failed horribly. Unfortunately this is a
common problem, not a theory. And the larger the project,
the more code there is to review, the more likely people are
to take that stance and hide the whole thing in your well-
famous "SEP-field".

Originally Posted by Tinkster Or: There's so many people who will be checking this, why should I bother. Now if "everyone" thinks like that the many eyes have failed horribly. Unfortunately this is a common problem, not a theory. And the larger the project, the more code there is to review, the more likely people are to take that stance and hide the whole thing in your well- famous "SEP-field".

But the fact is that in the *nix world there ARE a lot of people looking.

Generally it isn't a security expert working code to find problems to either report or exploit, it is the ordinary user, or the sysadmin, using the code for his own purposes and running across a discrepancy and reporting it. He may or may not recognize the discrepancy as a security issue, but the maintainer will of course recognize it when he discovers the problem.

I myself have reported bugs in linux code and applications, and never because I am trying to crack the code, but only because I have encountered the discrepancy. In all cases I have found the maintainers to be very responsive and cooperative.

And, lets go the next step. Have you yourself ever had a fully hardened system successfully cracked from outside?

I have not, though I have had them under attack time after time. Myself, I ALWAYS deploy a server with a small NAT router in front of it. This goes for both Linux and Windows, and no one at the NOCs where I put my boxes ever says "boo" when I put a small home-quality linksys or 3Com router between the box and their backbone. Just the NAT capability by itself safeguards me against the vast majority of exploits that exist. Perhaps the router itself is vulnerable, but most of its code exists in firmware so its vulnerability is limited, and its presence stops most probes for weaknesses from even reaching my servers.

No, I didn't have a break-in (to the best of my knowledge), but that's
not the point of this argument?

I doubt that a NATing home router device will protect you from SQL injects,
cross-site scripting or buffer overrun exploits. And the fact that there's
tons of vulnerability reports for packages (for whatever reason) is a good
indicator that the many eyes FAIL if the thing makes it into the wild and
then requires a patch. I'm not saying that this is any different for
closed-source apps, I'm saying that OpenSource and "many eyes" is not a silver
bullet, and that one can't be paranoid enough. And that relying on that idea
is what gets people pwned. Turning a blind eye (there's those eyes again) to
this is the first step to disaster. Glad you were lucky so far =}


Cheers,
Tink

Originally Posted by Tinkster No, I didn't have a break-in (to the best of my knowledge), but that's not the point of this argument? I doubt that a NATing home router device will protect you from SQL injects, cross-site scripting or buffer overrun exploits. And the fact that there's tons of vulnerability reports for packages (for whatever reason) is a good indicator that the many eyes FAIL if the thing makes it into the wild and then requires a patch. I'm not saying that this is any different for closed-source apps, I'm saying that OpenSource and "many eyes" is not a silver bullet, and that one can't be paranoid enough. And that relying on that idea is what gets people pwned. Turning a blind eye (there's those eyes again) to this is the first step to disaster. Glad you were lucky so far =} Cheers, Tink

Certainly a NAT router won't protect from SQL injects or cross-site scripting. It does provide some significant protection against buffer overruns.

And the vulnerability reports for packages...are in fact the evidence of the eyes working.

SQL injects occur in all cases because of careless programming, commonly in PHP scripts where user input is being trusted, not tested. Most commonly this is the result of amateurs programming, not a bug due to a more subtle issue. Yes, of course it is a problem, but it is a different kind of problem. And, for instance, mod_security (a package that I absolutely HATE, even as I recognize the need in hosting environments where there are ignorant amateurs programming) provides a solution to that. The servers I deploy are not vulnerable to this because I am not deploying web servers, though I do work on web servers.

Cross-site scripting is a mostly a client-side problem, and can be eliminated server-side by simply validating the user input and denying inappropriate text strings. Once again, this is handled at the user level and not at the system level. Yes, of course it is a problem nonetheless, but by definition a server that has user-written code running on it that was done incorrectly is not fully locked down.

If you are running web servers and selling space to clients, then locking down your server is close to impossible. If you are running any kind of server, and retaining control of all code that faces the net, then you certainly should be able to make the server so close to invulnerable that the difference barely matters.

This doesn't mean you can ignore that server; security is a process and not a destination. But if you have full control and know what you are doing, and deny anyone physical access to the machine, you can secure it nearly perfectly.

I have Windows 2000 servers that have been facing the net for years now, and have come under steady attack just like every net-facing server is attacked, and have never been broken. I control them remotely via SSH, VNC tunneled through SSH, and via a dedicated control panel that talks to my dedicated servers through a dedicated protocol.

And my Linux servers? No one has even come close.

But then, I am not selling space to clients for webhosting.

And the websites I support? One was cracked once, shortly after I took over support, and after the client refused to fund the security upgrades I wanted to put in place. After the site was cracked, he gave me carte blanche.

In my early days of programming PHP, an early-generation form I deployed was cracked and used to relay spam. I learned of that quickly, and studied the problem, and deployed solutions, and no one has cracked any of my sites since.