Please remember that saying "it didn't work" is not an unambiguous account of actions as it does not explain if it is the installation you're talking about or running Atop. If we're talking installation then Atop should be searched for in the RPMForge repo, and for yum to recognize a new repository you have to install the .repo file first by installing the rpmforge-release rpm from http://rpmforge.net/.


Reviewing your post history with us, and with all due respect, I kind of doubt that because of this, this and this.

Sure you have been trying to do this, this and this, but you also did do this and that.

So I doubt your list is complete. And if it really is complete then, due to how you indicated handling things, I doubt you have taken all necessary precautions. There is a percentage of GNU/Linux users who think that having a web-based panel equals having basic and administrator knowledge, and that being able to use a web-based panel for server administration releases them from the obligation to do more than the panel allows them to. Please be careful about what you do as much as how you do it.


Are you saying that because of how your machine is set up? Or are you saying that because you ran manual or automated tests to confirm and ensure it is that way?


Most services will dump logs in /var/log by default unless configured otherwise. Minimally check /var/log/messages, /var/log/secure, /var/log/http.* and all the other daemon logs. If you really deployed this instead of some sort of "grep-some-log-and-email-me-results" kludge then it should by default cover those logs. Also note Logwatch can be run again over all available logs and archived logs if such a report would be needed.


Running 'file' on those entries will show a clue about their function. By default the sess_.* are PHP session data caches, /tmp/.ICE-unix is created by startx (meaning you did ran or run a GUI, which isn't good or necessary on a headless machine), lost+found is a filesystem default directory, there's a MySQL socket and a directory in which spamd drops its initialization files.


See your /etc/http.* and /etc/http.*/.* include files.


OK. Here's a script you could easily cronjob wrt system activity. It doesn't do much except maybe show clues:
if you don't trust or otherwise rather not cronjob things you can run the simple version manually when high load occurs:
Please attach the result .log file to your next post.

You are right, I should have been more specific. Sorry. This is what I did and where I stopped.



I will not take offense to someone telling me I am wrong, when I am wrong. Especially someone trying to help me.

Wasn't sure this was what was asked, but I set this up to allow connections from my own IP address so one of my boxes here at home can run daily database backups at 3AM.

I thought I did mention this. It's installed, but is not running right now. I was at the beginning, but I since shut it down and the load problem still happens. So I didn't think it was part of the problem, thus didn't need to be listed.

All of that stuff is installed with a default WMH/Cpanel install. I'm not sure exactly what all services are running with it. Should have been more specific and I do apologize.

I'm not even sure I got that installed correctly. I do see it listed in the processes every so often when I'm looking at them, but whether or not it's doing it's job is beyond me.

I did not do this, yet. I assumed the issue with the load might have been multiple connections from clients, but I'm not so sure this is the problem.

I only wanted to know this, because it seems some bot is uploading .htaccess files to my pubic FTP server, not something that is needed and it appears to be a "bad" file intended to do harm. I simply delete it every time I see it and just ban the IP from the entire server now.

I set this back. I only did this to install the bf2 demo server a while back, I think. I know it was something I was trying to install and wanted to use the tmp directory, but my server wouldn't allow it. I attempted to change the path in which the installer wanted to use as tmp, but it failed the MD5 hash. So I just temporarily removed noexe from the /tmp directory.

This was on a site that I host for a friend, the problem has gone away but I have no idea what was causing it. I think the problem was caused by someone obtaining his password to his FTP, as some other files were being altered on his account. Once I changed his password, everything has been running fine since. I have also been working with him to get his site upgraded to Phpbb 3, but something is going on in with his life right now that he isn't able to accomplish much. I really want to show him how to do this, instead of just doing it myself like I usually do, because.. he wont learn anything.

You've just described me. Sadly, but I want to learn how to do things properly. Usually when things start messing up on my previous dedicated servers, I just migrate to a new one. That has defiantly not taught me anything.



I recall trying to gain access to those without logging in and each time it denied my access.


I have logwatch setup to email me all logs, granted I don't receive very many of them. I don't notice anything that "doesn't" appear to belong, but with my untrained eyes. I really have no idea.



So those files are essentially, normal files?



/usr/local/apache/htdocs appears to be the main root directory for apache. There are no unusual files in here though.

400.shtml
401.shtml
403.shtml
404.shtml
500.shtml
cp_errordocument.shtml
index.html
ea3_apache_build_htdocs/index.html
suspended.page/index.html

I have a bunch of web sites hosted on the server, each one has it's own document root directory and none of those files look out of place either, but I'll double check with them again.

I setup the cronjob to run every 30 minutes, I also ran the manual version (server load was not high) but nothing seemed to happen. Meaning, I opened a document in my root's home directory, called script.sh, I then copied the code to this document using pico and then saved the document. I CHMOD'd it with +x permissions then ran ./script.sh. No files were created and nothing was displayed in the terminal. Just a new line for a new command. I assume it only creates data when the server load is high?

Please install the rpmforge-release rpm from http://rpmforge.net/ first.


Actually a good thing, making backups, but since it runs over TCP now did you restrict access to MySQL in your firewall? BTW you can also avoid public connections by setting up an SSH tunnel from a machine at home to the server and run the connection over that. Speaking of SSH, you don't allow root logins if you run SSH, right?


If it's installed properly check the process commandline args ('pgrep -l snort') or /etc/snort/snort.conf to see what gets logged where. Usually /var/log/snort/. If logging in text mode (slower than binary) then there's log reporting tools for Snort. If logging in binary mode you'll want 'barnyard' to make logs human readable.


Any chance of posting the contents of one of those .htaccess files?


If you like paying for dead trees maybe get something like "Red Hat Enterprise Linux 5 Administration Unleashed" else see http://rute.2038bug.com/rute.html.gz and http://www.centos.org/docs/5/ and http://tldp.org/ and http://www.linuxquestions.org/linux/answers/ and http://www.howtoforge.com/. Together they should cover the basics you want and need.


Good!


Most network attacks will be preceded by lots of scanning. If something breaks it'll often be logged after that kind of "noise" (loads of 404 log lines).


That is an assumption. Using standard tools like 'stat' 'strings', 'file' and 'fuser' you can verify they are harmless.


/etc/cron.d/runonce.cron, if you correctly saved it as such and set its /etc/crontab entry, will only fire if loadavg is 10 or over but the "simple" script will run always. The reason you're not seeing anything is they're logging to /var/log: run 'ls -lrt /var/log/chk_*.log' and you'll see. Attach one from the cronjob as soon as it appears, OK?

Sweet, I got it installed now.

Oh boy. I don't have a firewall installed on my box. I just use iptables. No, I did not restrict access to MySQL. As far as I know, cPanel does this automatically. This is why I had to give my IP access rights to be able to access it. Previously, if I didn't give it the access rights though cPanel it would not allow me to connect.



They are stored in /var/log/snort/ when I open the files, I'm not sure what I am looking at. There are some strange characters followed by noraml characters mixed throughout the file. Here is an example:



Absolutely.


I have no problem buying and reading books. Are you recommending that I buy all of these and read them or do they all cover the same thing?


Yea I don't see anything like that in the logs I get. The only 404s I get usually people misspelling something, something that got moved on the web server and/or something most browsers automatically look for, such as the icon images.



So I ran the command on all those files and the general return string was either ASCII text, with no line terminators or empty.



I ran ls -lrt /var/log/chk_*.log and got /bin/ls: /var/log/chk_*.log: No such file or directory returned to me. I will defiantly attach one of the logs from the cronjob.

I apologise, in advance, for what I am about to do; the factual part of this post would have been much better, if it had been posted earlier, but I am afraid that I didn't see the thread when it originally started. I am doing this as much in the hope that it will help for the future, as anything.

You are right, to the extent that very high wait numbers are problematic, but not necessarily a good pointer to the cause of the problem.

Essentially, there is no information here; read the man page on vmstat and you will see that the first line doesn't help you:

because the information since the last reboot can 'dilute' the problem, the critical data can just disappear with this dilution. You would actually need to run continuously, for a while, and look at everything except that first line.

In addition, with a problem like this, if the 'bad stuff' peaks up and down, you could just miss it if you don't sample over a period of time.

To get any worthwhile information out of vmstat (if there is any to be had) you need to run it something like 'vmstat 2 10' (10 lines of data, each line covering 2 seconds...you'd probably want to run for rather longer than that) to give repeated data, ignore the first line of data, and to run over a sufficient period of time, so that brief, but severe, peaks in loading would get caught. Remember, if there was one brief, but very severe, peak in loading a high load queue value could persist for some time afterwards.

Just as a matter of interest (and of eliminating a long shot), can you say something about the disk subsystem - is it a completely conventional, single disk, subsystem, or is there something more complex going on there (like, eg, a raid array or a remote disk subsystem, such as SAN).

Is it clear that you can't have something indexing your disk drive?

For investigating i/o-related system loads, iotop is a good tool.

....b****y h*** that's a radical, and very cheering, attitude


Just a warning about these panel thingies, then; from my very limited experience of panels, they do make life dramatically simpler, because you don't have to bother about much of the detail. The trouble is that bothering about much of the detail, or at least understanding the detail and knowing which parts of the detail are critical and which aren't, is the bit that can cut off security issues before they start.

I've heard good things about cpanel, but only from website designers, who cannot be relied upon to know the first thing about security, so, from my point of view, caution is still indicated.

That may not sound like much -after all, you are not, I hope, a three letter agency, or the Mr Slim's personal banker, so why would a hacker go after you? Well, partly because they can and because they don't know what they could get, until after they have taken a can opener to your security measures. And, for some hacks, any box that they can use to anonymise their attack is pretty much as good as any other...

iptables is Linux's firewall system. Of course, there is a massive difference between a well-crafted set of iptables rules and just having 'gone with the default', with more-or-less everything open, because closing things up 'causes problems'.


Those were normal, if slightly surprising filenames. unSpawn is trying to go through with you whether the content of those files indicates anything to worry about.

I'll just pick up on
if I may.

Someone (you, or someone else) has tried to run a GUI. That would not be a normal thing to do if you were running a remote server. If you know that at some time you made the mistake of doing this, it may not be a specific worry. However, if you know that you have never tried to run a gui on this machine, you would need to get very concerned about the 'or someone else' aspect, particularly as the 'someone else' would, essentially, have to have been past all of your security measures in order to do this.

So, it is a file you would normally see on a desktop machine, it is not a file that you would normally see on this kind of server. There may be an innocent explanation of this, there may be a very, very guilty explanation of this, the filename, by itself, doesn't tell you that.

In principle, looking at the file meta info (date) might give you a clue; trouble is, if a hacker has got that far, they would have probably replaced utilities like 'ls' with hacked versions, and you would, therefore, not be able to trust their output. OTOH, there is no sign (yet?) of this being a competent hack attempt...

I do recall trying to get the GUI to work, by logging into it remotely via GUI interface. I'm just not 100% sure if it was this particular box or the previous one. I also don't remember what all I had attempted to do, as this was most likely when I got the box. How can I check to see if someone, other than myself, or even find out who attempted to get a GUI started on the box?

Redundant ...

This is not 100% reliable, but, if the date stamp on the file corresponds to a time at which you think you may have tried to run the GUI, there is quite a good chance that it was you.

If the logs stretch back that far, and they may easily not do that, look at any IP addresses that were used; is there anything odd there (assuming that an IP that you would used is non-odd...you may not have a fixed IP for the box that you would habitually used for access, but that should be within a limited range).

A general comment
This thread is becoming a bit scattergun; wandering off on all sorts of 'could be' tracks. This may get to an answer in the longer term, but, it seems to me, it would stand a better chance if particular ideas were thoroughly investigated and either demonstrated to be true, or not.

There have been a number of 'post the output of...' questions, not all of which have got the answer that would have been expected. Without those answers, it is unlikely that anyone will be able to say 'we have eliminated possibility A, therefore it has got to be B or C'.

There is a high wait time, a long queue and poor response times; it seems that these are related (and it would make sense for them to be related), but I have no idea whether this is a consequence of expected and obvious processes consuming more resources than expected or devious and hidden processes consuming resources. These are very different problems, and your next step would be very different depending upon which of those is true.

1 members found this post helpful.

I agree and I am waiting for the script to create logs. So far, the server's load hasn't been abnormal.

Also the date stamp on that directory .ICE-Unix was for March-12, which is NOT within' the time I've had the server. I've only had it since like June/July of last year. So that would mean a few days ago, which can't possibly be true. Unless the GUI is still "running" in a sense. Also when I change directory to that directory, it's empty.

Little update on this, for those of you who have offered advice.

I started over with Apache and rebuilt it with options that my scripts use. So far, I have not had a single over load since then. Not sure if it's just been a coincidence or if I did solve the problem however. Though it's been a couple weeks now with no problems. The highest I've noticed the load average is right around .30 now. Which is beautiful.