LinuxQuestions.org

LinuxQuestions.org (http://www.linuxquestions.org/questions/index.php)
-   Linux - Security (http://www.linuxquestions.org/questions/forumdisplay.php?f=4)
-   -   Help me my server was hacked (http://www.linuxquestions.org/questions/showthread.php?t=4175436804)

ctamayoa 11-12-2012 12:22 PM

Help me my server was hacked
 
I have a zentyal server, and today I got a message from hotmail "Your domain is bloked for spam activities"
Checking on my auth.log I could see this message
ebox : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/var/lib/zentyal/tmp/UKipoqrRQ4.cmd
I'm not using root as a user, is this means that my server have been hacked?
Also I dont understand the .cmd files on /var/lib/zentyal/tmp/

Please any help is welcome.

unSpawn 11-12-2012 01:03 PM

Quote:

Originally Posted by ctamayoa (Post 4827727)
I'm not using root as a user, is this means that my server have been hacked?
(..)I dont understand the .cmd files on /var/lib/zentyal/tmp/

Not necessarily: "TTY=unknown" means the process doesn't have a terminal attached like a user would have on login, "PWD=/" could denote an automated process like a cron daemon and /var/lib/zentyal/tmp/ seems to be a directory for temporary files.

It wouldn't hurt looking at what services your machine runs (web server, email, FTP and other servers), their logs and user login records. Do you have shell access via SSH or do you only use the web-based management panel?

ctamayoa 11-12-2012 01:10 PM

Thanks a lot for your help, I can access to the server by SSH and by Web-Base-Management (Zentyal control panel), but both are only accesible from local network.
But a russian hacker is still sending spamming from my server.

Quote:

Originally Posted by unSpawn (Post 4827748)
Not necessarily: "TTY=unknown" means the process doesn't have a terminal attached like a user would have on login, "PWD=/" could denote an automated process like a cron daemon and /var/lib/zentyal/tmp/ seems to be a directory for temporary files.

It wouldn't hurt looking at what services your machine runs (web server, email, FTP and other servers), their logs and user login records. Do you have shell access via SSH or do you only use the web-based management panel?

Also why in that log it shows as root I have no user root only administrator

unSpawn 11-12-2012 02:07 PM

Quote:

Originally Posted by ctamayoa (Post 4827751)
I can access to the server by SSH and by Web-Base-Management (Zentyal control panel), but both are only accesible from local network.

Good precaution.


Quote:

Originally Posted by ctamayoa (Post 4827751)
But a russian hacker is still sending spamming from my server.

OK, tell us what your web server provides like Joomla or another CMS, Wordpress, maybe a forum or statistics package, that kind of software, look at your standard syslog and mail server log files for clues, as root run the lastlog, last and who commands to check who has accessed your server and list the files in /var/lib/zentyal/tmp/.


Quote:

Originally Posted by ctamayoa (Post 4827751)
Also why in that log it shows as root I have no user root only administrator

"root" is a standard account in UNIX. It is not (or should not be) used as a user account, only for system maintenance.

ctamayoa 11-12-2012 02:26 PM

Quote:

Originally Posted by unSpawn (Post 4827796)
Good precaution.


Sorry I'm wrong I have no CMS no webpages are located in my server.
The only web app is the default control panel of Zentyal.
Can I post some logs here please?

unSpawn 11-12-2012 03:46 PM

Quote:

Originally Posted by ctamayoa (Post 4827804)
The only web app is the default control panel of Zentyal.

So what purpose do you use the server for? Is it a mail server or does it share files? What?


Quote:

Originally Posted by ctamayoa (Post 4827804)
Can I post some logs here please?

Sure.

While you're at it also please run these commands as root (it's one line):
Code:

( \ps axfwwwe 2>&1; lsof -Pwln 2>&1; netstat -antTupe 2>&1; lastlog 2>&1; last 2>&1; who -wai 2>&1; find /tmp /var/tmp /usr/tmp /var/spool/cron /var/lib/zentyal/tmp -printf "%T@ %A@ %C@ %u %g %m %y \"%p\"\n" 2>&1 ) > /tmp/output.log
then check the plain text file "/tmp/output.log" for any IP addresses you need to obscure, then attach the file to your reply.

ctamayoa 11-13-2012 12:05 AM

Quote:

Originally Posted by unSpawn (Post 4827866)
So what purpose do you use the server for? Is it a mail server or does it share files? What?



Sure.

While you're at it also please run these commands as root (it's one line):
Code:

( \ps axfwwwe 2>&1; lsof -Pwln 2>&1; netstat -antTupe 2>&1; lastlog 2>&1; last 2>&1; who -wai 2>&1; find /tmp /var/tmp /usr/tmp /var/spool/cron /var/lib/zentyal/tmp -printf "%T@ %A@ %C@ %u %g %m %y \"%p\"\n" 2>&1 ) > /tmp/output.log
then check the plain text file "/tmp/output.log" for any IP addresses you need to obscure, then attach the file to your reply.

Thanks a lot for your help my friend.
I did just what you said but I could'n find an ip that doesn't belong to our network.
Here I uploaded "output.log"
I also uploaded the mail.log file
Can you check them out please?
I can't figure out how my server is used for spamming activities. What log do I have to check because I have no clue about this

But I can see that is still sending spamming because in spamhouse I'm getting this message.
IP Address 186.101.6.94 is listed in the CBL. It appears to be infected with a spam sending trojan or proxy.
It was last detected at 2012-11-12 17:00 GMT (+/- 30 minutes), approximately 13 hours ago.
This IP is infected (or NATting for a computer that is infected) with the cutwail spambot. In other words, it's participating in a botnet.

unSpawn 11-13-2012 08:30 AM

Quote:

Originally Posted by ctamayoa (Post 4828061)
Here I uploaded "output.log"

Thanks. It seems your "administrator" account does not have the rights you need: you have to be root or be able to run the commands as root. Since I noticed you run Postgrey I also added that as reporting tool. Please run this modified set of commands as root user:
Code:

( [ `id -u` -eq 0 ]|| exit 1; ps axfwwwe -opid,ppid,gid,uid,cmd 2>&1; lsof -Pwln 2>&1; netstat -antupe 2>&1; zcat /var/log/maillog.*.gz 2>/dev/null; cat /var/log/maillog | postgreyreport −−nosingle_line −−check_sender=mx,a −−separate_by_subnet="=net=\n" 2>&1; find /var/www /var/spool -printf "%T@ %A@ %C@ %u %g %m %y \"%p\"\n" 2>&1 ) > /tmp/output1.log



Quote:

Originally Posted by ctamayoa (Post 4828061)
I can't figure out how my server is used for spamming activities.

That's what we're trying to figure out by reading the output you supply.

ctamayoa 11-13-2012 09:04 AM

Quote:

Originally Posted by unSpawn (Post 4828401)
Thanks. It seems your "administrator" account does not have the rights you need: you have to be root or be able to run the commands as root. Since I noticed you run Postgrey I also added that as reporting tool. Please run this modified set of commands as root user:
Code:

( [ `id -u` -eq 0 ]|| exit 1; ps axfwwwe -opid,ppid,gid,uid,cmd 2>&1; lsof -Pwln 2>&1; netstat -antupe 2>&1; zcat /var/log/maillog.*.gz 2>/dev/null; cat /var/log/maillog | postgreyreport −−nosingle_line −−check_sender=mx,a −−separate_by_subnet="=net=\n" 2>&1; find /var/www /var/spool -printf "%T@ %A@ %C@ %u %g %m %y \"%p\"\n" 2>&1 ) > /tmp/output1.log


Thanks a lot again for your time, maybe if before running this command line I type sudo, because I don't know how else I can run it as a root.
I mean like this:
Code:

sudo ( [ `id -u` -eq 0 ]|| exit.........

unSpawn 11-13-2012 09:24 AM

Ah, OK. First save the command line to a file, say "/tmp/cmnds".

Method 0: at. Then check if the "at" service is running. Then run
Code:

sudo at -f /tmp/cmnds now
This should run the "/tmp/cmnds" as root immediately.

Method 1: cron. Then check if the "cron" service is running. Check if /etc/cron.d directory exists. Then run
Code:

date +'%M %H * * * root /bin/bash /tmp/cmnds 2>&1 && rm -f /etc/cron.d/cmds.cron' --date="+2 minutes" > /tmp/cmnds.cron
sudo cp /tmp/cmnds.cron /etc/cron.d/

That should run /tmp/cmnds as root in 2 minutes, giving the cron daemon the time to pick up new cron jobs.

Method 2: shell. Run
Code:

sudo /bin/bash /tmp/cmnds
That should run /tmp/cmnds as root immediately as well.


*Note output will appear in "/tmp/output1.log", which is owned by root, so run
Code:

sudo chown administrator.administrator /tmp/output1.log
to gain ownership or
Code:

sudo chmod 0644 /tmp/output1.log
to gain just access rights.

Noway2 11-13-2012 01:31 PM

Quote:

Originally Posted by ctamayoa (Post 4828430)
Thanks a lot again for your time, maybe if before running this command line I type sudo, because I don't know how else I can run it as a root.

Sounds like you may be running Ubuntu or the root account has otherwise been locked. You may be able to get a root shell by using 'sudo -i'

@unSpawn: neat little command, postgreyreport. I've been using Postgrey for 4 years and never noticed it before.

ctamayoa 11-14-2012 12:17 AM

Quote:

Originally Posted by unSpawn (Post 4828443)
Ah, OK. First save the command line to a file, say "/tmp/cmnds".

Method 0: at. Then check if the "at" service is running. Then run
Code:

sudo at -f /tmp/cmnds now
This should run the "/tmp/cmnds" as root immediately.

Method 1: cron. Then check if the "cron" service is running. Check if /etc/cron.d directory exists. Then run
Code:

date +'%M %H * * * root /bin/bash /tmp/cmnds 2>&1 && rm -f /etc/cron.d/cmds.cron' --date="+2 minutes" > /tmp/cmnds.cron
sudo cp /tmp/cmnds.cron /etc/cron.d/

That should run /tmp/cmnds as root in 2 minutes, giving the cron daemon the time to pick up new cron jobs.

Method 2: shell. Run
Code:

sudo /bin/bash /tmp/cmnds
That should run /tmp/cmnds as root immediately as well.


*Note output will appear in "/tmp/output1.log", which is owned by root, so run
Code:

sudo chown administrator.administrator /tmp/output1.log
to gain ownership or
Code:

sudo chmod 0644 /tmp/output1.log
to gain just access rights.


Really Thank you my friend.
I just did it.
Unfortunately I can't upload the log here because its size is too big.
Here I uploaded the output.log
http://mailing-ecuador.info/output1.log

By the way in the report of spamhouse says:
This IP is infected (or NATting for a computer that is infected) with the cutwail spambot. In other words, it's participating in a botnet.

So I guess some computer in my local network is infected CUTWAIL spambot, but I need to find which one is.
I would like to know if there a way to log all outgoing emails with postfix.

unSpawn 11-14-2012 08:12 AM

Quote:

Originally Posted by ctamayoa (Post 4828902)
Here I uploaded the output.log

Thanks, unfortunately your postgreyreport failed:
Code:

Can't open ..nosingle_line: No such file or directory at /usr/bin/postgreyreport line 184.
Can't open ..check_sender=mx,a: No such file or directory at /usr/bin/postgreyreport line 184.
Can't open ..separate_by_subnet==net=\n: No such file or directory at /usr/bin/postgreyreport line 184.

Please check your postgreyreport man page on how to run it.


Quote:

Originally Posted by ctamayoa (Post 4828902)
So I guess some computer in my local network is infected CUTWAIL spambot, but I need to find which one is.

If this is Cutwail, you have two things on your hands: Pushdo (the distributor) and Cutwail (the spam-sending module). Here's the technical write-up of the affected vendor: Win32/Cutwail and here's what CBL: Scanning your machine for exploits and Mynetwatchman: Isolating infected hosts hidden behind NAT routers/firewalls/proxies write about detection in general.


Quote:

Originally Posted by ctamayoa (Post 4828902)
I would like to know if there a way to log all outgoing emails with postfix.

It already does: see /var/log/maillog. If you want more than that, like complete email content logging you have to read the documentation as it contains examples:
http://www.postfix.org/FILTER_README.html
http://www.postfix.org/ADDRESS_VERIFICATION_README.html
http://www.postfix.org/postconf.5.html#always_bcc
http://www.postfix.org/postconf.5.html#sender_bcc_maps
http://www.postfix.org/postconf.5.ht...pient_bcc_maps


Back to the local network infection I'll ask a few questions so I better understand things:
* By approximation, how long has this infection been going on?
* Have you had earlier cases of LAN machines being infected and what did you do at that time?
* How many machines are in your LAN and how many of those run Windows?

I'll outline in short what you should do:
0. Perform log analysis of all network and related log files in /var/log including Squid, Dansguardian and ClamAV and their rotated logs (as in "/var/log/dansguardian/access.log.*.gz"). Include bwmonitor logs in your analysis, maybe the amount of traffic or timing may provide clues,
1. Remove all white listing in Dansguardian because all in- and outbound network traffic must be scanned,
2. Review your firewall rules so no traffic passes in or out without filtering,
3. Remove all white listing in Postgrey (/etc/postgrey/whitelist_* ?) because all in- and outbound mail traffic must be subject to filtering,
4. Snort IDS seems to be supported in Zentyal so install it. While the current rules only consider Cutwail Command and Control URI's (Pushdo A, B and C because D encrypts traffic) it's better than nothing. Get the rule set from the Snort website, disable rules for services you don't run or that are otherwise outside of the scope (like deleted, Oracle, SCADA, VoIP, etc, etc). The rule set you must keep is botnet-cnc.rules. Then get the rule set from the Emerging Threats web site and disable along the same lines. The rule you must keep is called emerging-current_events.rules,
5. Scan every LAN machine with what you know from the "Scanning your machine for exploits" document.

Noway2 11-14-2012 08:30 AM

Quote:

Originally Posted by unSpawn (Post 4829175)
Thanks, unfortunately your postgreyreport failed:
Code:

Can't open ..nosingle_line: No such file or directory at /usr/bin/postgreyreport line 184.
Can't open ..check_sender=mx,a: No such file or directory at /usr/bin/postgreyreport line 184.
Can't open ..separate_by_subnet==net=\n: No such file or directory at /usr/bin/postgreyreport line 184.

Please check your postgreyreport man page on how to run it.

This is what happens when you try to run postgreyreport with sudo. You need to run it from a room prompt. I tried this yesterday on an Ubuntu host and got this exact same result, line number 184 and everything, which is what prompted me to make the post about using sudo -i to get the root shell.

ctamayoa 11-16-2012 12:23 AM

Quote:

Originally Posted by unSpawn (Post 4829175)
Thanks, unfortunately your postgreyreport failed:
Code:

Can't open ..nosingle_line: No such file or directory at /usr/bin/postgreyreport line 184.
Can't open ..check_sender=mx,a: No such file or directory at /usr/bin/postgreyreport line 184.
Can't open ..separate_by_subnet==net=\n: No such file or directory at /usr/bin/postgreyreport line 184.

Please check your postgreyreport man page on how to run it.



If this is Cutwail, you have two things on your hands: Pushdo (the distributor) and Cutwail (the spam-sending module). Here's the technical write-up of the affected vendor: Win32/Cutwail and here's what CBL: Scanning your machine for exploits and Mynetwatchman: Isolating infected hosts hidden behind NAT routers/firewalls/proxies write about detection in general.



It already does: see /var/log/maillog. If you want more than that, like complete email content logging you have to read the documentation as it contains examples:
http://www.postfix.org/FILTER_README.html
http://www.postfix.org/ADDRESS_VERIFICATION_README.html
http://www.postfix.org/postconf.5.html#always_bcc
http://www.postfix.org/postconf.5.html#sender_bcc_maps
http://www.postfix.org/postconf.5.ht...pient_bcc_maps


Back to the local network infection I'll ask a few questions so I better understand things:
* By approximation, how long has this infection been going on?
* Have you had earlier cases of LAN machines being infected and what did you do at that time?
* How many machines are in your LAN and how many of those run Windows?

1) This infection is approximately two and a half weeks
2) Is the first time I see a critical virus on my lan, it happens before but with little pen drive viruses, I always have updated my Nod 32 antivirus and running in every machine, if is still the problem I delete it manually.
3) I have 20 machines in my Lan with Windows

Quote:

Originally Posted by unSpawn (Post 4829175)
I'll outline in short what you should do:
0. Perform log analysis of all network and related log files in /var/log including Squid, Dansguardian and ClamAV and their rotated logs (as in "/var/log/dansguardian/access.log.*.gz"). Include bwmonitor logs in your analysis, maybe the amount of traffic or timing may provide clues,
1. Remove all white listing in Dansguardian because all in- and outbound network traffic must be scanned,
2. Review your firewall rules so no traffic passes in or out without filtering,
3. Remove all white listing in Postgrey (/etc/postgrey/whitelist_* ?) because all in- and outbound mail traffic must be subject to filtering,
4. Snort IDS seems to be supported in Zentyal so install it. While the current rules only consider Cutwail Command and Control URI's (Pushdo A, B and C because D encrypts traffic) it's better than nothing. Get the rule set from the Snort website, disable rules for services you don't run or that are otherwise outside of the scope (like deleted, Oracle, SCADA, VoIP, etc, etc). The rule set you must keep is botnet-cnc.rules. Then get the rule set from the Emerging Threats web site and disable along the same lines. The rule you must keep is called emerging-current_events.rules,
5. Scan every LAN machine with what you know from the "Scanning your machine for exploits" document.

Finally I could run your command on root user and this time no errors are displayed:
Here I uploaded the new version of the output1.log
http://mailing-ecuador.info/output1.log
here is the white list of Postgrey
http://mailing-ecuador.info/whitelist_clients.log

Thanks for your time

Regards

Carlos


All times are GMT -5. The time now is 10:03 AM.