Server suddently uploads huge amounts of data
 

Hello,

I'm having a problem with a Centos server. Sometimes it is almost like it is getting attacked or maybe more correctly attacking others. While this happens the network is completely overfloded and jammed.

If you look at the attached image you can see that my throughput for the server suddently rises enormously. The server is running a bit under 1000 websites as well as ftp for those. A few other services run in the background too. However I am not able to identify the source of my problems nor where the data is sent to.

Ntop is configured only to listen to traffic intended for the server itself (Non Promiscuous Mode).

I cannot seem to find anything in the logs, but that is probably because I don't know what to look for. Anybody got an idea for how to identify the problem?

Best regards
Morten

Change Root Password, make sure you can't ssh into the machine as root. Also if I can remember correctly "check" the httpd.conf. See if there are any changes to it, like a share directory. I could be wrong, but sounds like someone has hacked you.

First of all please disregard any "advice" suggesting you should edit, install, remove or change anything on the machine as that could hamper analysis.


0. Please post:
- the release version, if it runs (on) any form of virtualization (VMware, XEN, etc), process partitioning (LXC, OpenVZ, etc),
- which services the machine itself provides including web-based management panels, statistics, web log, forum, shopping cart, plugins and other software, and including what the web sites run, their exact software versions and if that software was kept up to date,
- which logging, access restrictions is in place and what hardening was performed,
- if there have been earlier breaches or anomalies,
- since when and how regularly this problem manifests itself, and
- anything else tangible, factual and supported by data worth mentioning.

1. Please attach as plain text or post, preferably in [code]vBB tags[/code] the complete listings of these commands (preferably first cd to /dev/shm or else if you must /tmp):
Running all system and daemon logs through Logwatch with the args. The latter is best done by copying logs to a separate machine or workstation and running Logwatch from there.

2. Since you run Ntop, can you drill down to one period and show detailed stats by protocol, destination port, etc?

Also please ask specific questions before performing if necessary, please reply verbosely and do stay with the thread (subscribe?) until completion and reply as soon as possible when replies are posted.

Sorry I didn't aim to hamper anything, just was a thought. Just incase someone has gotten into the system, and changed someone changed something. I worded the sentence wrong there. Not my strongest skill.

Since you're quite new to LQ and since you did mention the word, just so you know the LQ Linux Security forum aims to be as fast and efficient as possible especially when faced with potential breaches of security. Those who practice Incident Response / sniper forensics here like to keep the SNR as low as possible and the information presented to be relevant, factual and complete. Those w/o IR practical knowledge, those who won't stay with a thread until completion and those who think about posting a "don't worry", "I think" or "I guess" type of one-liner best wait until one of the handlers arrives and if they can't resist the compulsion to post just point the OP to the (outdated but still useful) CERT Intruder Detection Checklist,
TIA.

I'll try to gather as much of the requested information as possible, posting it here tomorrow morning. However I can already tell you that I am unable to find information about all the websites on the server. They all run independent of each other and the clients are able to upload their own websites and files.

Based upon that we can probably asume that there is old unsecured software running on at least a dozen webhotels. However the apache-installation runs with a lot of prohibited functions (see attached text-file), so my hope is, that it is enough to encounter insecure websites. Tell me if you need me to dig deeper into finding information about these websites.

The directory /var/www is owned by root and all subsequent directories are owned by the ftp-user. More information will follow tomorrow.

Since I asked you about 16 questions I know it will take time to process it. We'll just have to wait.

MortenOnDebian, is there any update to this situation? Have you been able to obtain any of the requested data? Is the problem still occurring and have you been monitoring the situation?

At last I have been able to obtain the requested information. I've uploaded it to a website of mine - see links below.

The server is running on VMware version 4.1 Update 1 - The newest update. It only runs the following services:
NTOP, ProFTPd, MySQL og Apache

The problem first occured around the 4th of September. After 3 days it started occuring once a day for a whole week.
Then it faded down to once every/every second week. The attacks normally occur in the late afternoon/evening + once
in the morning.

As an example one of the attacks occured the 24th of september from 5:42 pm until 6:05 pm when the power to the server
was cut off.

I was unable to only get the logwatch information from the above timespan, so the appended log is generated with
the "--range All" parameter. I've tried the command following commands with the only result of seing information about
the range parameter:

Files:
http://download.wep.dk/linux/logwatch.log
http://download.wep.dk/linux/rpmvfy.log
http://download.wep.dk/linux/logfile

Please ask if I have missed something. I hope I will be able to respond much faster.

Regards
Morten

...it also runs Xorg font server, Sendmail, RPC services and Webmin.
SELinux appears to be disabled.
Apache seems to be a standard configuration with all unnecessary modules loaded, mod_security seems to be in use.


Please post:
- which access restrictions are in place,
- what hardening was performed (if any),
- and since you run Ntop, can you drill down to one period and show detailed stats by protocol, destination port, etc?


Thanks. Please obfuscate the servers canonical name and its IP address in the logwatch report and then use bzip2 to compress it. I'll then download and look at it. Due to the reboot (which you didn't mention earlier) process data may have less relevance but log data all the more.

I also forgot to mention that IPTables isn't running. As far as I know there are no access restrictions. Futhermore there have not been done any hardening of the server.

The compressed logwatch file can now be found at:
http://download.wep.dk/linux/logwatch.log.bz2

Attached are to images from ntop showing the used protocols during the attacks. I've attached two images from 24th of september as well as from the 2nd of october, bot showing the same pattern.

Thanks. After a quick glance:

Quite a lot uploaded that apparently is not downloaded via FTP...


Rearranging HTTP stats you see a clear division between what "regular" content is accessed and what you would not like to see as regular content. There are two accounts that upload ISO's (search for "avlinux") and the other one of them (search for "windows7_universal.iso") was reported back in 2010 for hosting a Chase Online bank phishing scam and I don't think he's an official "Photoshop CS5 Master Collection" reseller :-]

To link U/L / D/L's with said time frame of events happening 18(!) days ago grep your Proftp and Apache logs for the date of the 24th and grep for these two accounts. The output is probably better readable if you run it through a statistics reporter like webalizer.

Hmm, I must admit I cannot get webalizer to display stats for the old log-files for September. I have exported them to a different server where I just installed webalizer, however it will only display data for October. As I said there had been an attack on the 2nd of October as well, so I went exploring this.

Even though the avlinux-domain has been logged for 17 gigabytes of data for the first 10 days of October (according to webalizer), there was no activity in the period of the attack. Same applies to the other domain you referred to. Going through the logs manually concluded this for both the 24th of September and the 2nd of October

However looking at the ntop graph I posted above, there seems to be a lot of torrent traffic. I have absolutely no clue from where this comes, but I guess it might have something to do with my problems. Do you have any suggestions?

Edit: Added the webalizer-graph for the "avlinux"-domain. As shown only a bit of the traffic occurs during the 2nd of October, where one of the attacks happened

The fundamental problems we hit are the limited amount of logging, running no firewall and less reporting than expected.

The latter ties in to a certain agree with the first but where I said I did expect a detailed traffic report by source and destination hosts, ports, duration and transferred data size per incident. Ntop doesn't use OSI layer 7 classification AFAIK, instead it relies, similarly to nmap, on port mappings like /etc/services provide so you can't learn from that if traffic really is Bittorrent or not. Having a list of source and destination ports (listings scraped from its HTML reporting pages rather than graphs) means you can either correlate that with logging to you have or not. Having a list of source and destination hosts means you can see if there's certain hosts that cause this or if individual hosts traffic is low look for other causes.

Running without firewall means no protection from invalid traffic like bogons, wrong flags, wrong local service ports or traffic to remote services you should not allow, no rate limiting, no accounting of traffic and no logging. Since the incidents occur still w/o having found the cause we should work towards firewall rules that accomplish all of that. For starters please email me the output of and I'll write the rules.

Finally I don't know if you're running any SAR like Atop, dstat or collectl (Mark: please don't ;-p) but if we can't get a fix on things then having resource utilization nfo could hold clues.

When you talk about limited logging, you will for sure be glad when I tell you, that ntop doesn't display data about hosts and ports from before it was last started. So as the server has been forced to shut down, I cannot make ntop display any information about the source/destination of traffic during the attack.

All I can see is the amount of traffic. Maybe thats just some sort of misconfiguration from my side, but I cannot seem to find anything which changes this behaviour.

As the iptables-save command was not found, I have just executed the command without that part. The output can be found at http://download.wep.dk/linux/log.txt

For the 24th OK but not even for October 2nd?

Anyway, here's a script to generate a logging-only rule set. It doesn't do anything but drop invalid packets and log at what the script perceives is the amount of packets per second the host can take so we get an idea of what's a storm and what not. Round off the numbers if you will, check rule set for flaws, adapt if necessary and test before production use:

Sorry for not responding. I did not want to just implement your script without fully understanding every part of it - After all its a live production environment. However while I was reading up on iptables and your rules the attacks stopped. I have not noticed an attack since mid october. The solution is still unknown, but I appreciate your help.

Not responding is one thing but this thread unearthed quite a lot of problems with that server. Please tell us that reading up on iptables and implementing that script wasn't the only thing you did the past four months.

I guess you refer to the fact that it was quite hard to acquire logging information, as well as the fact that there are several unused extensions for apache etc. which is running. I have tried to trim down some of them, but hoonestly I have not done much more.

You listed your main problem as your machine "getting attacked or attacking others" and as a consequence of that your network connection clogging up completely.

You also indicated:
- not being able to get a proper network usage overview,
- not having any idea of what web sites actually run,
- not possessing or not being able to find system logging over the relevant period,
- not running a firewall (at that time),
- not having any access restrictions configured, and
- not having any hardening of the server done (+ SELinux was disabled).

During the course of this thread it became clear analysis was hampered by:
- insufficient log retention, and
- lack of detail in application of choice (Ntop).


I. Given the severity of the problem and the hiatus we exposed what are the formal, technical reasons for not changing or implementing the above?
II. With the above list of things to address, what changes do you propose to make?

I guess there really are no good excuses for not implementing more security as described.

I have installed vnstat and are looking at iftop as well. Both of them should give me a better overview of the current network situation as well as a log for the upcoming past.

I am aware that a well configured firewall should always be up and running, and that I should be more strict about which ports/programs it allows to pass through. I'm currently looking at which programs should be running and which shouldn't.

Talking about hardening and SELinux is a different matter. I have heard a lot of people complaining about SELinux causing more trouble than good. Resulting in many hours spend debugging until finally realising why. Of course this might be people which don't know what they are talking about, but that is one reason why I have not installed it or something simular (any suggestions?).

No, there isn't. If you (think you) have other priorities then by all means let somebody else take care of it. Because the fact the symptoms stopped does not mean your problem disappeared automagically. Tools like Vnstat, iftop or equivalent get you traffic overview stats but you can't drill down further than stream level (it doesn't point to users or applications), you can't set threshold alerts (though you could have a cron job parsing vnstat output) but more importantly these tools don't prevent or mitigate anything (for bandwidth shaping use iproute and iptables).

It's good you're trying to find out which applications should be running and which shouldn't. But what are you using to find out? And if you find such applications how are you going to prevent them from running?

And wrt SELinux it isn't something you'll be implementing at this stage without gaining basic user knowledge first and rigorous testing on a non-production machine (virtualization?). But even without SELinux there's enough you could do: make a list of security and performance risks, add a solution or fix to each, gauge which changes would have the biggest positive effect and then prioritize work.

I do understand your point about logging tools not being enough. However I believe they are neat to have, as they provide me with more knowledge about how all is linked together. And I know they should not stand alone :)

About running programs and processes I'm currently just watching the output of the ps command though a cronjob. However I guess a utility like ps-watcher could ease this process and make it more informative, but I have not look into that. Yet.

Sitting down with a pen and a piece of paper writing down security and performance risks might be a good idea. Also I hope to be able to learn more about any performance problems through the extra logging tools. Also considered installing file::monitor letting it watch and report changes to directories, but yes again it does not prevent anything.

I guess the first question you should ask yourself is "what is the problem, how will I know about it and what will I do to mitigate it?" Then check your standard toolkit. Netfilter has modules for accounting and if you search Sourceforge, Berlioz, Savannah.nongnu or the-site-formerly-known-as-Freshmeat you may find tools that make things easier, for example ipband only starts logging when a threshold is crossed and can send alerts.


Watching processes is good but note it won't tell you everything you want to know, so in the end it'll be a combination of process names or paths "/home/user/.mutt/apache -DSSL", names of files kept open "/usr/bin/perl /tmp/.favicon/.ico " or combinations "wget some.ser.ver/user/xhide", ports and connections "irc.somechatserver.net", listing newly created files (inotify?), logged anomalies, general and per-process resource usage and more.


Feel free to post your list once you're ready.


No, it wouldn't prevent anything. But if you have general system resource usage statistics in place (Atop, Dstat, Collectl, like that) and network resource monitoring and if you find the discipline to look at reports regularly and act on them then at least you have a chance nipping things in the bud.