LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Security
User Name
Password
Linux - Security This forum is for all security related questions.
Questions, tips, system compromises, firewalls, etc. are all included here.

Notices


Reply
  Search this Thread
Old 11-12-2012, 12:22 PM   #1
ctamayoa
LQ Newbie
 
Registered: Nov 2012
Posts: 9

Rep: Reputation: Disabled
Help me my server was hacked


I have a zentyal server, and today I got a message from hotmail "Your domain is bloked for spam activities"
Checking on my auth.log I could see this message
ebox : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/var/lib/zentyal/tmp/UKipoqrRQ4.cmd
I'm not using root as a user, is this means that my server have been hacked?
Also I dont understand the .cmd files on /var/lib/zentyal/tmp/

Please any help is welcome.
 
Old 11-12-2012, 01:03 PM   #2
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,415
Blog Entries: 55

Rep: Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600
Quote:
Originally Posted by ctamayoa View Post
I'm not using root as a user, is this means that my server have been hacked?
(..)I dont understand the .cmd files on /var/lib/zentyal/tmp/
Not necessarily: "TTY=unknown" means the process doesn't have a terminal attached like a user would have on login, "PWD=/" could denote an automated process like a cron daemon and /var/lib/zentyal/tmp/ seems to be a directory for temporary files.

It wouldn't hurt looking at what services your machine runs (web server, email, FTP and other servers), their logs and user login records. Do you have shell access via SSH or do you only use the web-based management panel?
 
1 members found this post helpful.
Old 11-12-2012, 01:10 PM   #3
ctamayoa
LQ Newbie
 
Registered: Nov 2012
Posts: 9

Original Poster
Rep: Reputation: Disabled
Thanks a lot for your help, I can access to the server by SSH and by Web-Base-Management (Zentyal control panel), but both are only accesible from local network.
But a russian hacker is still sending spamming from my server.

Quote:
Originally Posted by unSpawn View Post
Not necessarily: "TTY=unknown" means the process doesn't have a terminal attached like a user would have on login, "PWD=/" could denote an automated process like a cron daemon and /var/lib/zentyal/tmp/ seems to be a directory for temporary files.

It wouldn't hurt looking at what services your machine runs (web server, email, FTP and other servers), their logs and user login records. Do you have shell access via SSH or do you only use the web-based management panel?
Also why in that log it shows as root I have no user root only administrator

Last edited by unSpawn; 11-12-2012 at 01:59 PM. Reason: //Merge
 
Old 11-12-2012, 02:07 PM   #4
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,415
Blog Entries: 55

Rep: Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600
Quote:
Originally Posted by ctamayoa View Post
I can access to the server by SSH and by Web-Base-Management (Zentyal control panel), but both are only accesible from local network.
Good precaution.


Quote:
Originally Posted by ctamayoa View Post
But a russian hacker is still sending spamming from my server.
OK, tell us what your web server provides like Joomla or another CMS, Wordpress, maybe a forum or statistics package, that kind of software, look at your standard syslog and mail server log files for clues, as root run the lastlog, last and who commands to check who has accessed your server and list the files in /var/lib/zentyal/tmp/.


Quote:
Originally Posted by ctamayoa View Post
Also why in that log it shows as root I have no user root only administrator
"root" is a standard account in UNIX. It is not (or should not be) used as a user account, only for system maintenance.
 
1 members found this post helpful.
Old 11-12-2012, 02:26 PM   #5
ctamayoa
LQ Newbie
 
Registered: Nov 2012
Posts: 9

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by unSpawn View Post
Good precaution.

Sorry I'm wrong I have no CMS no webpages are located in my server.
The only web app is the default control panel of Zentyal.
Can I post some logs here please?

Last edited by unSpawn; 11-12-2012 at 03:15 PM. Reason: //Fix quote tag
 
Old 11-12-2012, 03:46 PM   #6
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,415
Blog Entries: 55

Rep: Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600
Quote:
Originally Posted by ctamayoa View Post
The only web app is the default control panel of Zentyal.
So what purpose do you use the server for? Is it a mail server or does it share files? What?


Quote:
Originally Posted by ctamayoa View Post
Can I post some logs here please?
Sure.

While you're at it also please run these commands as root (it's one line):
Code:
( \ps axfwwwe 2>&1; lsof -Pwln 2>&1; netstat -antTupe 2>&1; lastlog 2>&1; last 2>&1; who -wai 2>&1; find /tmp /var/tmp /usr/tmp /var/spool/cron /var/lib/zentyal/tmp -printf "%T@ %A@ %C@ %u %g %m %y \"%p\"\n" 2>&1 ) > /tmp/output.log
then check the plain text file "/tmp/output.log" for any IP addresses you need to obscure, then attach the file to your reply.
 
Old 11-13-2012, 12:05 AM   #7
ctamayoa
LQ Newbie
 
Registered: Nov 2012
Posts: 9

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by unSpawn View Post
So what purpose do you use the server for? Is it a mail server or does it share files? What?



Sure.

While you're at it also please run these commands as root (it's one line):
Code:
( \ps axfwwwe 2>&1; lsof -Pwln 2>&1; netstat -antTupe 2>&1; lastlog 2>&1; last 2>&1; who -wai 2>&1; find /tmp /var/tmp /usr/tmp /var/spool/cron /var/lib/zentyal/tmp -printf "%T@ %A@ %C@ %u %g %m %y \"%p\"\n" 2>&1 ) > /tmp/output.log
then check the plain text file "/tmp/output.log" for any IP addresses you need to obscure, then attach the file to your reply.
Thanks a lot for your help my friend.
I did just what you said but I could'n find an ip that doesn't belong to our network.
Here I uploaded "output.log"
I also uploaded the mail.log file
Can you check them out please?
I can't figure out how my server is used for spamming activities. What log do I have to check because I have no clue about this

But I can see that is still sending spamming because in spamhouse I'm getting this message.
IP Address 186.101.6.94 is listed in the CBL. It appears to be infected with a spam sending trojan or proxy.
It was last detected at 2012-11-12 17:00 GMT (+/- 30 minutes), approximately 13 hours ago.
This IP is infected (or NATting for a computer that is infected) with the cutwail spambot. In other words, it's participating in a botnet.

Last edited by unSpawn; 11-13-2012 at 07:01 AM. Reason: //Removed attachments
 
Old 11-13-2012, 08:30 AM   #8
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,415
Blog Entries: 55

Rep: Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600
Quote:
Originally Posted by ctamayoa View Post
Here I uploaded "output.log"
Thanks. It seems your "administrator" account does not have the rights you need: you have to be root or be able to run the commands as root. Since I noticed you run Postgrey I also added that as reporting tool. Please run this modified set of commands as root user:
Code:
( [ `id -u` -eq 0 ]|| exit 1; ps axfwwwe -opid,ppid,gid,uid,cmd 2>&1; lsof -Pwln 2>&1; netstat -antupe 2>&1; zcat /var/log/maillog.*.gz 2>/dev/null; cat /var/log/maillog | postgreyreport −−nosingle_line −−check_sender=mx,a −−separate_by_subnet="=net=\n" 2>&1; find /var/www /var/spool -printf "%T@ %A@ %C@ %u %g %m %y \"%p\"\n" 2>&1 ) > /tmp/output1.log



Quote:
Originally Posted by ctamayoa View Post
I can't figure out how my server is used for spamming activities.
That's what we're trying to figure out by reading the output you supply.
 
Old 11-13-2012, 09:04 AM   #9
ctamayoa
LQ Newbie
 
Registered: Nov 2012
Posts: 9

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by unSpawn View Post
Thanks. It seems your "administrator" account does not have the rights you need: you have to be root or be able to run the commands as root. Since I noticed you run Postgrey I also added that as reporting tool. Please run this modified set of commands as root user:
Code:
( [ `id -u` -eq 0 ]|| exit 1; ps axfwwwe -opid,ppid,gid,uid,cmd 2>&1; lsof -Pwln 2>&1; netstat -antupe 2>&1; zcat /var/log/maillog.*.gz 2>/dev/null; cat /var/log/maillog | postgreyreport −−nosingle_line −−check_sender=mx,a −−separate_by_subnet="=net=\n" 2>&1; find /var/www /var/spool -printf "%T@ %A@ %C@ %u %g %m %y \"%p\"\n" 2>&1 ) > /tmp/output1.log

Thanks a lot again for your time, maybe if before running this command line I type sudo, because I don't know how else I can run it as a root.
I mean like this:
Code:
sudo ( [ `id -u` -eq 0 ]|| exit.........

Last edited by unSpawn; 11-13-2012 at 09:25 AM. Reason: //Fix quotes *again*???
 
Old 11-13-2012, 09:24 AM   #10
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,415
Blog Entries: 55

Rep: Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600
Ah, OK. First save the command line to a file, say "/tmp/cmnds".

Method 0: at. Then check if the "at" service is running. Then run
Code:
sudo at -f /tmp/cmnds now
This should run the "/tmp/cmnds" as root immediately.

Method 1: cron. Then check if the "cron" service is running. Check if /etc/cron.d directory exists. Then run
Code:
date +'%M %H * * * root /bin/bash /tmp/cmnds 2>&1 && rm -f /etc/cron.d/cmds.cron' --date="+2 minutes" > /tmp/cmnds.cron
sudo cp /tmp/cmnds.cron /etc/cron.d/
That should run /tmp/cmnds as root in 2 minutes, giving the cron daemon the time to pick up new cron jobs.

Method 2: shell. Run
Code:
sudo /bin/bash /tmp/cmnds
That should run /tmp/cmnds as root immediately as well.


*Note output will appear in "/tmp/output1.log", which is owned by root, so run
Code:
sudo chown administrator.administrator /tmp/output1.log
to gain ownership or
Code:
sudo chmod 0644 /tmp/output1.log
to gain just access rights.

Last edited by unSpawn; 11-13-2012 at 09:26 AM. Reason: //In case the file isn't executable prefix shell
 
Old 11-13-2012, 01:31 PM   #11
Noway2
Senior Member
 
Registered: Jul 2007
Distribution: Gentoo
Posts: 2,125

Rep: Reputation: 781Reputation: 781Reputation: 781Reputation: 781Reputation: 781Reputation: 781Reputation: 781
Quote:
Originally Posted by ctamayoa View Post
Thanks a lot again for your time, maybe if before running this command line I type sudo, because I don't know how else I can run it as a root.
Sounds like you may be running Ubuntu or the root account has otherwise been locked. You may be able to get a root shell by using 'sudo -i'

@unSpawn: neat little command, postgreyreport. I've been using Postgrey for 4 years and never noticed it before.

Last edited by Noway2; 11-13-2012 at 01:35 PM.
 
Old 11-14-2012, 12:17 AM   #12
ctamayoa
LQ Newbie
 
Registered: Nov 2012
Posts: 9

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by unSpawn View Post
Ah, OK. First save the command line to a file, say "/tmp/cmnds".

Method 0: at. Then check if the "at" service is running. Then run
Code:
sudo at -f /tmp/cmnds now
This should run the "/tmp/cmnds" as root immediately.

Method 1: cron. Then check if the "cron" service is running. Check if /etc/cron.d directory exists. Then run
Code:
date +'%M %H * * * root /bin/bash /tmp/cmnds 2>&1 && rm -f /etc/cron.d/cmds.cron' --date="+2 minutes" > /tmp/cmnds.cron
sudo cp /tmp/cmnds.cron /etc/cron.d/
That should run /tmp/cmnds as root in 2 minutes, giving the cron daemon the time to pick up new cron jobs.

Method 2: shell. Run
Code:
sudo /bin/bash /tmp/cmnds
That should run /tmp/cmnds as root immediately as well.


*Note output will appear in "/tmp/output1.log", which is owned by root, so run
Code:
sudo chown administrator.administrator /tmp/output1.log
to gain ownership or
Code:
sudo chmod 0644 /tmp/output1.log
to gain just access rights.

Really Thank you my friend.
I just did it.
Unfortunately I can't upload the log here because its size is too big.
Here I uploaded the output.log
http://mailing-ecuador.info/output1.log

By the way in the report of spamhouse says:
This IP is infected (or NATting for a computer that is infected) with the cutwail spambot. In other words, it's participating in a botnet.

So I guess some computer in my local network is infected CUTWAIL spambot, but I need to find which one is.
I would like to know if there a way to log all outgoing emails with postfix.

Last edited by ctamayoa; 11-14-2012 at 12:56 AM. Reason: Added more text
 
Old 11-14-2012, 08:12 AM   #13
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,415
Blog Entries: 55

Rep: Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600
Quote:
Originally Posted by ctamayoa View Post
Here I uploaded the output.log
Thanks, unfortunately your postgreyreport failed:
Code:
Can't open ..nosingle_line: No such file or directory at /usr/bin/postgreyreport line 184.
Can't open ..check_sender=mx,a: No such file or directory at /usr/bin/postgreyreport line 184.
Can't open ..separate_by_subnet==net=\n: No such file or directory at /usr/bin/postgreyreport line 184.
Please check your postgreyreport man page on how to run it.


Quote:
Originally Posted by ctamayoa View Post
So I guess some computer in my local network is infected CUTWAIL spambot, but I need to find which one is.
If this is Cutwail, you have two things on your hands: Pushdo (the distributor) and Cutwail (the spam-sending module). Here's the technical write-up of the affected vendor: Win32/Cutwail and here's what CBL: Scanning your machine for exploits and Mynetwatchman: Isolating infected hosts hidden behind NAT routers/firewalls/proxies write about detection in general.


Quote:
Originally Posted by ctamayoa View Post
I would like to know if there a way to log all outgoing emails with postfix.
It already does: see /var/log/maillog. If you want more than that, like complete email content logging you have to read the documentation as it contains examples:
http://www.postfix.org/FILTER_README.html
http://www.postfix.org/ADDRESS_VERIFICATION_README.html
http://www.postfix.org/postconf.5.html#always_bcc
http://www.postfix.org/postconf.5.html#sender_bcc_maps
http://www.postfix.org/postconf.5.ht...pient_bcc_maps


Back to the local network infection I'll ask a few questions so I better understand things:
* By approximation, how long has this infection been going on?
* Have you had earlier cases of LAN machines being infected and what did you do at that time?
* How many machines are in your LAN and how many of those run Windows?

I'll outline in short what you should do:
0. Perform log analysis of all network and related log files in /var/log including Squid, Dansguardian and ClamAV and their rotated logs (as in "/var/log/dansguardian/access.log.*.gz"). Include bwmonitor logs in your analysis, maybe the amount of traffic or timing may provide clues,
1. Remove all white listing in Dansguardian because all in- and outbound network traffic must be scanned,
2. Review your firewall rules so no traffic passes in or out without filtering,
3. Remove all white listing in Postgrey (/etc/postgrey/whitelist_* ?) because all in- and outbound mail traffic must be subject to filtering,
4. Snort IDS seems to be supported in Zentyal so install it. While the current rules only consider Cutwail Command and Control URI's (Pushdo A, B and C because D encrypts traffic) it's better than nothing. Get the rule set from the Snort website, disable rules for services you don't run or that are otherwise outside of the scope (like deleted, Oracle, SCADA, VoIP, etc, etc). The rule set you must keep is botnet-cnc.rules. Then get the rule set from the Emerging Threats web site and disable along the same lines. The rule you must keep is called emerging-current_events.rules,
5. Scan every LAN machine with what you know from the "Scanning your machine for exploits" document.
 
Old 11-14-2012, 08:30 AM   #14
Noway2
Senior Member
 
Registered: Jul 2007
Distribution: Gentoo
Posts: 2,125

Rep: Reputation: 781Reputation: 781Reputation: 781Reputation: 781Reputation: 781Reputation: 781Reputation: 781
Quote:
Originally Posted by unSpawn View Post
Thanks, unfortunately your postgreyreport failed:
Code:
Can't open ..nosingle_line: No such file or directory at /usr/bin/postgreyreport line 184.
Can't open ..check_sender=mx,a: No such file or directory at /usr/bin/postgreyreport line 184.
Can't open ..separate_by_subnet==net=\n: No such file or directory at /usr/bin/postgreyreport line 184.
Please check your postgreyreport man page on how to run it.
This is what happens when you try to run postgreyreport with sudo. You need to run it from a room prompt. I tried this yesterday on an Ubuntu host and got this exact same result, line number 184 and everything, which is what prompted me to make the post about using sudo -i to get the root shell.
 
2 members found this post helpful.
Old 11-16-2012, 12:23 AM   #15
ctamayoa
LQ Newbie
 
Registered: Nov 2012
Posts: 9

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by unSpawn View Post
Thanks, unfortunately your postgreyreport failed:
Code:
Can't open ..nosingle_line: No such file or directory at /usr/bin/postgreyreport line 184.
Can't open ..check_sender=mx,a: No such file or directory at /usr/bin/postgreyreport line 184.
Can't open ..separate_by_subnet==net=\n: No such file or directory at /usr/bin/postgreyreport line 184.
Please check your postgreyreport man page on how to run it.



If this is Cutwail, you have two things on your hands: Pushdo (the distributor) and Cutwail (the spam-sending module). Here's the technical write-up of the affected vendor: Win32/Cutwail and here's what CBL: Scanning your machine for exploits and Mynetwatchman: Isolating infected hosts hidden behind NAT routers/firewalls/proxies write about detection in general.



It already does: see /var/log/maillog. If you want more than that, like complete email content logging you have to read the documentation as it contains examples:
http://www.postfix.org/FILTER_README.html
http://www.postfix.org/ADDRESS_VERIFICATION_README.html
http://www.postfix.org/postconf.5.html#always_bcc
http://www.postfix.org/postconf.5.html#sender_bcc_maps
http://www.postfix.org/postconf.5.ht...pient_bcc_maps


Back to the local network infection I'll ask a few questions so I better understand things:
* By approximation, how long has this infection been going on?
* Have you had earlier cases of LAN machines being infected and what did you do at that time?
* How many machines are in your LAN and how many of those run Windows?
1) This infection is approximately two and a half weeks
2) Is the first time I see a critical virus on my lan, it happens before but with little pen drive viruses, I always have updated my Nod 32 antivirus and running in every machine, if is still the problem I delete it manually.
3) I have 20 machines in my Lan with Windows

Quote:
Originally Posted by unSpawn View Post
I'll outline in short what you should do:
0. Perform log analysis of all network and related log files in /var/log including Squid, Dansguardian and ClamAV and their rotated logs (as in "/var/log/dansguardian/access.log.*.gz"). Include bwmonitor logs in your analysis, maybe the amount of traffic or timing may provide clues,
1. Remove all white listing in Dansguardian because all in- and outbound network traffic must be scanned,
2. Review your firewall rules so no traffic passes in or out without filtering,
3. Remove all white listing in Postgrey (/etc/postgrey/whitelist_* ?) because all in- and outbound mail traffic must be subject to filtering,
4. Snort IDS seems to be supported in Zentyal so install it. While the current rules only consider Cutwail Command and Control URI's (Pushdo A, B and C because D encrypts traffic) it's better than nothing. Get the rule set from the Snort website, disable rules for services you don't run or that are otherwise outside of the scope (like deleted, Oracle, SCADA, VoIP, etc, etc). The rule set you must keep is botnet-cnc.rules. Then get the rule set from the Emerging Threats web site and disable along the same lines. The rule you must keep is called emerging-current_events.rules,
5. Scan every LAN machine with what you know from the "Scanning your machine for exploits" document.
Finally I could run your command on root user and this time no errors are displayed:
Here I uploaded the new version of the output1.log
http://mailing-ecuador.info/output1.log
here is the white list of Postgrey
http://mailing-ecuador.info/whitelist_clients.log

Thanks for your time

Regards

Carlos

Last edited by ctamayoa; 11-16-2012 at 01:24 AM. Reason: More text
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Server hacked bruxelles2010 Linux - Security 9 11-15-2010 07:23 AM
Server hacked cpanelskindepot Linux - Security 46 07-05-2004 06:19 PM
Server hacked php4u Linux - Security 1 07-05-2004 11:34 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Security

All times are GMT -5. The time now is 01:02 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration