Hi leslie..

This is a saple output of my pcap file:
Does this mean anything to you ???

255.255.255.255 is my server ip number.

I'd remove your IP from that because you've got a compromised server and you don't want to advertise that.

All you capture shows there is a load of ack's. You need to let it run for a while, checking your sendmail logs to see if you've got the line of interest. Once you have we can open up that capture file in Wireshark and hopefully find the culprit.

For security don't post your capture file here - or any contents of it. Just post back when you've caught the activity in a capture file and I'll be happy to try and guide you through it.

I've done a quick port scan of your IP (you'll probably see me in your logs if you look) and you don't have anything really untoward open there other than this: 10000/tcp open snet-sensor-mgmt I'll tell you now I see that from time to time, but I've never looked into it. It seems to talk HTTP and gives this header: Server: MiniServ/1.540 but I'm not sure if it's anything to do with you or something Zen are doing.

This is Webmin

Ah. Thanks. Mmmm. It may be worth checking the log file for that too - to see if anyone had made it in past that.

It would appear that the user, altys-of, gained a shell prompt and that they logged in from the University of Central Lancashire. From a little Google searching: The question then becomes was this user account, or your FTP process exploited for this purpose? If this user was not supposed to be able to have a command shell, I would give it some extra close attention.

I do recognize most of the processes listed in your output. Most legitimate processes will show a normal path and parent process chain. Look very closely for ones that are spawned from a user that should not be creating those processes, such as an FTP user or Nobody. You are correct that the LSOF command does provide a list of open files and processes and should also show you who owns them. What you are looking for are processes that are owned by unexpected users, bogus processes owned by root, and processes that are tied to hidden files and files in temporary locations. For the processes you don't recognize, Google can often times be your friend. For example, migration/0, migration/1. I couldn't tell you what they are, but they do show up in lists as being normal and the /0, /1 seems to indicate association with the CPU core (dual core processor). Since your running Centos, you should also be able to use the RPM verify command to make sure that your binaries have not been compromised or modified.

Since you are having an outgoing SPAM problem, it is likely that there is a hidden process running on your system somewhere that is causing these messages to appear. The trick will be to use the list of processes and open files to find what is causing them to appear. This is where the log files, LSOF, and PS commands come into play and the benefit, and hard part too, is to cross correlate the entries. Since you have text of the message, you might want to try giving the strings command or grepping for files that contain some keywords from the SPAM. Once you find the source of the SPAM, you will need to determine how they gained entry. Unless the log files have been tampered with, which indicates a root level compromise, there should be lots of noise associated with this. I would look for either a lot of SSH or FTP connection failures, or strange process failures, like segmentation faults that could indicate that the process was crashed.

1 members found this post helpful.

Please post your thread once and in only one forum. Posting a single thread in the most relevant forum will make it easier for members to help you and will keep the discussion in one place. Your threads in the Linux Security forum of the same topic have been merged, hence the merge haiku for those who get the reference:

Eye close to the glass,
Horrified he sees the fly:
Merger Successful.

0 members found this post helpful.

Hi Noway2...

Thanks for getting back to me.
As it happens, my techie in America (Moon) managed to stop the abuse last night at about 2:00AM.
Here is what he had to say:
and
also
Unfortunately, he is a Professor in a University in America and is very limited on time.
Linux is a hobby to him.

You have tried to help me and I am very grateful for all your thoughts and suggestions.

I wonder what your thoughts are to the above.
I'm afraid it is a little above my head !!!

I believe that I still need help to actually find the 'way in' in the first place.

I still can't get it out of my mind that:

gained shell access.
I've now changed his access to NOLOGIN

I've had my ISP on the phone this morning telling me that they have received more complaints.
I was lucky enough to be able to tell him that the problem is now over. At least for the time being until we manage to 'nail' the problem completely.
During the conversation, he did suggest that entry is sometimes gained via Wordpress web sites.
Co-incidently, altys-of-preston.co.uk web site is designed with Wordpress.
It could be a loophole, but I wouldn't know where to start looking for it.
It does seem to be a coincidence that I had no problems until I took him as a client.

Please let me know your thoughts.
The history behind altys-of-preston.co.uk (which is registered to me) web site is that some IT students from Lancashire University had to do a project to design a commercial web site for a genuine business.
They approached Alty's Of Preston, a motor mechanic dealing in Volvo and Saab cars, and offered to design, create and manage a brand new web site completely free of charge as part of their project. The designed the web site with Wordpress, which I believe is reputed to have some security issues.

I don't know how they gained a shell access. When I set their user account up I gave them /bin/ftponly acount.
I have now changed this to NOLOGIN acount and changed their password until I can get to the bottom of the problem.

I REALLY DO APPRECIATE ALL YOUR HELP.

THANK YOU

I am glad that you have found our assistance helpful. It looks like you have found the cause of the spam. The two big questions remain: how did they gain entry and how badly were they able to penetrate your system. Here are some more questions and things to check into for your investigation.

1 - Regarding the TLS SMTP authentication, would you please clarify as to what leads them to this conclusion? Specifically, what evidence is there in the log files to suggest this. One possibility would be a weak password on a user account that had sufficient permissions to call sendmail from the local system.

2 - It does look like they managed to gain a shell prompt using this user. I would one, run some tests to see if an FTP user like you had them configured would be able to get this type of access. Once they had a shell prompt, they could conceivably use WGET to retrieve the file and place it on your system. Other possibilities include, remote file inclusion exploits on Apache, or loss of containment on your FTP site. What version of these programs are you running. Also what is the patch level. The same goes for Wordpress, which I am assuming you are also using PHP. What is it's rev level. Once you have the rev level, you will want to check the CVE listings to see if there are known exploits for these programs.

3 - The hidden perl script is important. What user and permissions were associated with the file. Specifically, was it a root level directory and or file. If there is evidence of root level compromise?

4 - How was the script being called? Was there a modification in the cron table or some other mechanism. If it was via a remote command, the netstat output should provide you with some important information.

5 - The person doing your investigation said the following: What leads them to this conclusion. Again, there should be some form of evidence to support this.

If you haven't already, and assuming you haven't been root level compromised, your log files should contain noise related to the intrusion attempt. Try running your log files through logwatch to look for clues. At this point, it will be important for you to determine if they compromised a user account only or if they were able to gain root. Finding evidence related to their activities will provide insight into this.

Hi Noway2..

Many thanks for getting back to me.
I was hoping that you would.
Many thanks for such a comprehensive reply.
However, for the main part - I'm lost !!!
I couldn't agee more - BUT HOW ???
I have no idea what led Moon (my techie) to come to this conclusion. Over the many years that he has helped me, he has always been a man of few words.
I can't check the log files because they rotate every 4 weeks. Since the apparent intrusion was on the 13th October it was more than 4 weeks ago.
No user login has shell access except for ukzone (me) and moon (my techie), and this is only temporary during this problem.
All other users have either NO LOGIN or FTPONLY
Therefore, other than myself, nobody has permissions to to call sendmail other than via a script.
I wouldn't know where to start testing this. Maybe some guidance may help.
My FTP server is: vsftpd-2.0.5-16.el5_5.1
Server version: Apache/2.2.17 (Unix)
My PHP: php-5.2.16-jason.1
My mysql: mysql-5.1.52-jason.1
Wordpress: I don't know. This was setup by altys web design team.
I don't know the answer to this. I did ask: How did you KILL the perl script if you didn't know where and what it was ??? and I haven't had an answer as yet. I wouldn't normally expect one until tomorrow morning. He is in the USA and I am in the UK.
I dont know how the script was being called. Maybe if we knew what the file was we may be able to see how it was being called.
I did check the cron table and it was just as I had set it up.
I believe that NETSTAT is a real time command. If so, I cannot check this since the problem has now stopped.
Since I am on a final warning from my ISP, I wouldn't dare ask Moon to 'turn it on' again.
I don't know. I don't understand it so I didn't ask. He might have told me and I would be even more confused.
I know that you're trying to help and you must think that I'm an absolute dummy, but I wouldn't know where to start with this.

I'm pretty sure that the script doesn't come from any web site.
Yesterday I deleted ALL the web sites (after copying them) in batches. I waited for about 30 minutes and when the email flow didn't stop I returned the web sites.
I concluded that if I setup a new server nd transferred all the sites I would be OK.
As a matter of fact, if the abuse hadn't stopped when it did I was going to have to do just that.

Once again, I can't thank you enough for all the help and advise that you have given me, even if I don't fully understand.
I manage to cope with running a web server under normal circumstances. But when I get a serious problem like this - I'M LOST !!!

Once again,

THANK YOU

Lets take things one at a time. First, I am a little confused and would ask that you please clarify. Are you currently having a spam problem or does it appear to have been stopped. What is confusing me are your two statements: and If you are currently (still) experiencing an outgoing spam problem, especially after the script file was supposedly stopped /deleted, it says that you have some underlying compromise. If you are continuing to have problems, you should really consider taking this system off line. If you have an operator with physical access, you can pull the network cable, otherwise you should use the iptables firewall to restrict connections to SSH from a trusted source(s) only.

As far as how to conduct an intrusion investigation, the CERT checklist gives a good outline of the steps you should follow. Lets take them one step at a time.
1 - examine the log files for unusual activity. Unfortunately, this may be hampered by the loss of log information due to your rotate, but it may still turn up information. You should manually look through the logs, looking for error entries such as failed password attempts, and various error messages like segmentation faults. You already found good information through this process, locating a user who apparently had shell access. Have Moon send you the logs and then run the file set through the tool logwatch with the detail set to high and the full time range. Here is an example command: 2 - Look for setuid and setgid files (especially setuid root files)everywhere on your system. These would be files that are owned by a user such as root and are set to execute with root level privilege. It could also be a file set to execute with another user's privilege. The checklist gives you the find commands to run to do this.
3 - Check your system binaries to make sure that they haven't been altered. Since your running CentOS you can make use of the RPM -vV command. The -v is for verbose output and the -V is a verify. This will verify the signature of your files against the RPM database. I would also use the find command to look for any files that have been modified within the last 5 weeks or so.
4 - Check your systems for unauthorized use of a network monitoringprogram, commonly called a sniffer or packet sniffer. This can be accomplished through the use of the LSOF and netstat commands. Note, I saw your concern about netstat. Netstat is a passive command that shows you the connection (port) status on your machine. You may be thinking of nmap which is an active scanning program.
5 - Examine all the files that are run by 'cron' and 'at.' In other words, check all of the cron files, crontab for each user, cron.daily, etc. Quoting from unSpawn in this thread (may also be of interest to you): 6 - Check for unauthorized services. Examine your Apache, PHP, and wordpress configurations very closely, especially if they have been modified within or around the time frame of interest.
7 - Examine the /etc/passwd file. Also look at your shadow file. Has it been modified? Unfortunately, you may have tampered with this one when trying to change the status of the FTP user.
7- Check your system and network configuration files for unauthorized entries.
8 - Look everywhere on the system for unusual or hidden files. You will want to make a more detailed look for these. If you are still having problems, something is still hidden somewhere, and or you have an open connection to the cracker (netstat output).

Other things to consider. When running these tests, you might want to obtain known good binaries from the CentOS repository. If you have been root level compromised, you could have Trojan horse files that will give you false results.

Looking at the version information you have posted:
1 vsftpd-2.0.5. Version 2.0.5 is dated to August 2006. The age of this program, and the fact that it is about 8 versions behind is a MAJOR RED FLAG. Off hand it looks like vsftpd's big vulnerability is for a DOS, not privilege escalation.
2 Apache 2.2.17: Released October 18, 2010. This isn't too bad. The current version is .21 though you will be somewhat dependent on your distribution for this. Looking a the known vulnerabilities, I don't see anything obvious.
3 PHP 5.2.16 was released 16 December 2010. Same thing with Apache, not too bad. PHP, though does open you to a wide range of vulnerabilities. I would pay particular attention to what you are running on it, e.g. wordpress. I would find out what version your running and review the release history for known exploits.
4 mysql-5.1.52 was also released late in 2010. This is probably not, by itself the culprit.

Hope the above helps. I realize that it is a lot of information. It is important for you to go through the steps and obtain as much information as best you can. At this point, I would make my priority to determine if you have a root level compromise. If you do, you are looking at a system loss and your recovery efforts will likely be in vain. If you need help with interpreting and analyzing the information, contact either myself or unSpawn via email or private message and we can make arrangements to get you some (free) assistance in this regard.

1 members found this post helpful.

Hi Noway2...

This is just a quick reply to explain the situation.

I'm very sorry if I haven't explained myself clearly.

First of all, the SPAM has stopped due to:
This was 2:00AM on Monday morning.

This was referring to the fact that BEFORE Moon managed to stop the spam. I deleted ALL web sites in batches, weaited for 30 minutes and the spam did not stop.
I then returned the web sites satisfied that the spam was not coming from any web site.

The current situation is that the spam has stopped.
BUT the cause has not been found.

I will update more later...

The fact that the spam has stopped is potentially a good sign. If you can identify the script that sent the spam, along with its file permissions, location, and what was calling it, this will tell you a lot about the degree of the compromise. If it looks like the script was in a world writable location, such as /tmp, or you had a non privileged user account compromised without being able to escalate to root, you have a chance of saving the system, bringing things up to date and taking preventative measures. I am still a little concerned about the FTP only user getting a shell prompt, however, given the age of your FTP program and the availability of tools like metasploit, this remains an avenue to investigate.

2 members found this post helpful.