Real Forensics Case
 

so, i have a rhel 5.7 x86_64 that had some IRC stuff on it. and, the system looks like it was doing dictionary attack of SSH to lots of public systems. main concern right now is to determine if any SS#'s were moved off the system.

i am using clamscan along with some cmd line find & grep & regex looking in all ascii files for patterns.

any suggestions on what to do and what not to do, or tools ?

Yes. You're doing more anti-forensics than forensics. Stop it and get professional help.

how so? i am trained in forensics, just not using Encase or FTK for this incident.

Best first thing: do nothing. Nothing but read, think and decide on an approach. Please first read the Intruder Detection Checklist (CERT): http://web.archive.org/web/200801092...checklist.html. It may be old but the steps are still usable.

Before you would go in and Do Stuff you should inventory 0) what you have already in terms of off-site backups and remote syslog, 1) what data (login records, logs, live process data) you need to secure and if 2) you should perform bit-by-it copies if you suspect deletion, tampering or if ample time has passed between compromise and discovery (so check available storage capacity as well).

The goal is to gather enough information to 0) build an understanding of the situation, 1) allow you to mitigate and preserve (or start over) and 2) draw conclusions. By building an understanding of the situation I mean you saying "some IRC stuff" implies the machine is deployed in a 'net-facing role and saying "SS#'s" suggests e-payment, but there's more than that. (BTW if it's subject to regulations ensure you fully understand the consequences of hiring professional forensics firm vs the DIY approach.)

What we'd like to know:
- purpose of the server (web server, proxy, etc),
- when you discovered it, what clues you found,
- which actions you performed (killing processes, deleting files, etc),
- what audit data, user login records, system, daemon and firewall logs you already secured,
- details of running services, processes and network connections,
- existence of any setuid root files or foreign objects in temp dirs, docroot and users homes,
- user shell history files.
* and any nfo you think could help.

As for live data listing please use something like Move all data to a separate workstation you'll use for log analysis etc and run all system and daemon logs through Logwatch with these args: Finally please:
- attach plain text (if any) or if too large email me or Noway2 to discuss a drop-off,
- post results from the CERT Intruder Detection Checklist too,
- ask questions, reply verbosely and stay with the thread until completion.

unspawn,
the system is post hands-on by numerous people. i am doing post hands-on work looking for evidence that SS#'s were moved off the system. i have evidence that there was a IRC C2 on this system and was likely being used for a SSH dictionary attack. the IRC was quarantined by other people. this system is not in use at this time.

SS#'s were in Oracle db (a RAC node) on this system. i know the regs, hence why i am trying to prove SS#'s did or did not leave this system. the IRC made an outbound JOIN connection and thats where we think the C2 was, and shortly after we see lots of SSH connections outbound, and we have file that is a dictionary full of uid: password

but right now i need to only look for SS#'s that are outside of the db, or evidence that the db was queried directly by unauthorized source and the #'s moved off the box, etc.

If you're into forensics and Encase/FTK, why aren't you posting on the Encase/AccessData or other forensic forums?

Some things to check...

Since you say the computer is down, I take it memory wasn't imaged?
Is there a bash history of value?
Is there indication the attacker had the appropriate rights to get the SSNs?
Is there any database logging?
What about other logging?
Have you created a timeline?
Any evidence that the attacker was even aware of the database?
Was the data encrypted?
Are there compressed files that contain user data?
Is there any indication of malware being capable of transferring the data?
Large netflows that suggest data was sent out?
Any DLP software used?
Do you have to comply with data breach notification laws?
If so I believe if you can't determine if the data was compromised, you won't have to notify all clients if you can pinpoint a time of compromise. Check with a lawyer though. :)

the system is not down, its not being used, taken out of the cluster. i will take a mem dump to file.

i am not using Encase or FTK for this work, just trying to search the live system for SS#'s, etc

not sure if attacker knew of the db. db had PII that is not encrypted

we have legal involved, so disclosure is being handled appropriately

Don't take this the wrong way, but your actions seem more like a sysadmin than someone in forensics. The questions you're asking also suggest you're not experienced enough to be handling this, and it doesn't appear as though you're doing this in a forensically sound manner...

In general you collect the volatile data, pull the plug and image the drive (or if you have to, image while the computer is running). You seem to be running AV scans, searching for SSNs and doing your investigation on a live computer. At what point are you planning on imaging the drive, because I think most people in forensics would say imaging should have been done a while ago. Are you even documenting your actions? If it's just some server then that's one thing, but things change a bit when you're responsible with peoples identities...

AFAIK he's not running clamscan for AV purposes but because it has a SSN filter. And while examiners do tend to club data and drag it off to their lair that doesn't mean in situ Live examinations don't take place. For instance during a seizure a party may ask for a rough confirmation of suspicions or an examiner may be forced by whatever circumstance to make an informed decision on the spot which data to capture.


I'm not sure how I can be of any help. It seems you know about regular expressions, you should know where data hides and (I assert) you tested your tools and validated their output before using them in the field...

Sure, first responders need to do preliminary investigations all the time to confirm whether an incident happened, and need to collect data, but it definitely doesn't sound like he's in the preliminary stages. Investigating a live computer to determine whether it's compromised is one thing, but IMO, I think most in forensics would agree that determining what the attacker did and whether SSNs were compromised isn't something that should be done on a live computer... Especially with serious cases like this one apparently is.

what i am doing is very valid in the context of what my customer wants done.

and unSpawn - yes, i have been using various searching methods along with regex to search for SS# strings, along with some tools. i still need to do log analysis of various logs to see if the database itself was being queried in odd ways around the time we saw the IRC channel joins. i suspect a IRC C2 if commanded to move data via network it would not move into a file before being transferred, so i need to dump the RAM and take a look in there for SS # patterns, and if matching patterns found i then need to compare those against all of those in the db.

at this point i suspect the IRC compromise was not a direct vector to this system but rather from another compromised system in the environment (there are other issues elsewhere too). what i see from IDS logs is an indicative egress SSH brute force attack with a majority of it to 128.12.x.x, followed by 128.10.x.x and 128.11.x.x, and a few others, but what does seem odd is, i have infection file that is a dictionary for SSH logins, which seem odd if the IRC was a true C2 (C2's dont need files to do things, etc).


and for OlRoy, i do converse with folks on the Encase forums, and to ease your concern i do understand this forensics is not the norm, but its the customer's $$ and this is what they want me to do.

A lot of what you're saying isn't making sense to me, but I'm not going to dwell on it... Since there is an IDS, there may be a signature that looks for SSNs in clear text. If so, you should also look for those alerts, but keep in mind there could be false positives/negatives.

I'm not sure where you're getting the "C2 doesn't need files to do things" from. IRC bots and SSH dictionary attacks are pretty common, and there is nothing odd about having a file of logins/passwords and an IRC bot.

there is no need to drop files onto the system if i can get C2 via a IRC channel, thats my point. if its a C2 with files then its a poor C2. a shovel over tunnel is much stealthier. and no, my IDS did not see SS #'s, nor at this time do i know if the IDS is configured to alarm on clear text SS #'s. IDS is useless for encrypted or encoded data, etc.

What you're describing is extremely common linux malware and attack tools. C2 means Command and Control channel, which only describes the way an attacker controls a computer. It has nothing to do with not putting files on a computer. In fact, finding an IRC bot means you should expect to find more malicious files on a computer. Putting files on a computer is even often a wanted feature of "Command and Control". Even if the C2 isn't capable of putting files on a computer, or is capable and loads them directly in memory, there is no reason why the attacker couldn't also use SSH to manage the computer to put files on it. Bottom line is what you're describing is extremely common and shouldn't be odd to someone who is experienced in investigating compromises...

Knowing the IDS didn't give any SSN alerts doesn't mean much unless you know whether it's configured to look for them or not. And yes, like I said, there can be false positives/negatives, but it is a source of evidence that should be checked.

exactly what i said, i need to go check if IDS can alert on SS #'s.

and i'll say it again, a good C2 doesnt put files on the system. do you have forensics training or background ??