LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Linux - Security (http://www.linuxquestions.org/questions/linux-security-4/)
-   -   Server suddently uploads huge amounts of data (http://www.linuxquestions.org/questions/linux-security-4/server-suddently-uploads-huge-amounts-of-data-4175428963/)

MortenOnDebian 09-25-2012 11:04 AM

Server suddently uploads huge amounts of data
 
1 Attachment(s)
Hello,

I'm having a problem with a Centos server. Sometimes it is almost like it is getting attacked or maybe more correctly attacking others. While this happens the network is completely overfloded and jammed.

If you look at the attached image you can see that my throughput for the server suddently rises enormously. The server is running a bit under 1000 websites as well as ftp for those. A few other services run in the background too. However I am not able to identify the source of my problems nor where the data is sent to.

Ntop is configured only to listen to traffic intended for the server itself (Non Promiscuous Mode).

I cannot seem to find anything in the logs, but that is probably because I don't know what to look for. Anybody got an idea for how to identify the problem?

Best regards
Morten

2armz 09-25-2012 11:09 AM

Change Root Password, make sure you can't ssh into the machine as root. Also if I can remember correctly "check" the httpd.conf. See if there are any changes to it, like a share directory. I could be wrong, but sounds like someone has hacked you.

unSpawn 09-25-2012 11:44 AM

First of all please disregard any "advice" suggesting you should edit, install, remove or change anything on the machine as that could hamper analysis.


Quote:

Originally Posted by MortenOnDebian (Post 4788956)
I'm having a problem with a Centos server. (..)
The server is running a bit under 1000 websites as well as ftp for those. (..)
A few other services run in the background too. (..)

0. Please post:
- the release version, if it runs (on) any form of virtualization (VMware, XEN, etc), process partitioning (LXC, OpenVZ, etc),
- which services the machine itself provides including web-based management panels, statistics, web log, forum, shopping cart, plugins and other software, and including what the web sites run, their exact software versions and if that software was kept up to date,
- which logging, access restrictions is in place and what hardening was performed,
- if there have been earlier breaches or anomalies,
- since when and how regularly this problem manifests itself, and
- anything else tangible, factual and supported by data worth mentioning.

1. Please attach as plain text or post, preferably in [code]vBB tags[/code] the complete listings of these commands (preferably first cd to /dev/shm or else if you must /tmp):
Code:

( /bin/ps axfwwwe -opid,ppid,uid,context,cmd 2>&1; /usr/sbin/lsof -Pwln 2>&1; \
find /var/spool/cron 2>&1; /bin/netstat -anpe 2>&1; /usr/bin/lastlog 2>&1; /usr/bin/last \
-wai 2>&1; /usr/bin/who -a 2>&1 ) > logfile 2>&1

Code:

/bin/rpm  --nodeps --noscripts --notriggers -Vva 2>&1|/bin/grep -v "\.\{8\}" > rpmvfy.log 2>&1
Running all system and daemon logs through Logwatch with the
Code:

--detail High --service All --range All --archives --numeric --save logwatch.log
args. The latter is best done by copying logs to a separate machine or workstation and running Logwatch from there.

2. Since you run Ntop, can you drill down to one period and show detailed stats by protocol, destination port, etc?

Also please ask specific questions before performing if necessary, please reply verbosely and do stay with the thread (subscribe?) until completion and reply as soon as possible when replies are posted.

2armz 09-25-2012 12:02 PM

Sorry I didn't aim to hamper anything, just was a thought. Just incase someone has gotten into the system, and changed someone changed something. I worded the sentence wrong there. Not my strongest skill.

unSpawn 09-25-2012 12:19 PM

Quote:

Originally Posted by 2armz (Post 4789001)
skill.

Since you're quite new to LQ and since you did mention the word, just so you know the LQ Linux Security forum aims to be as fast and efficient as possible especially when faced with potential breaches of security. Those who practice Incident Response / sniper forensics here like to keep the SNR as low as possible and the information presented to be relevant, factual and complete. Those w/o IR practical knowledge, those who won't stay with a thread until completion and those who think about posting a "don't worry", "I think" or "I guess" type of one-liner best wait until one of the handlers arrives and if they can't resist the compulsion to post just point the OP to the (outdated but still useful) CERT Intruder Detection Checklist,
TIA.

MortenOnDebian 09-25-2012 12:59 PM

1 Attachment(s)
I'll try to gather as much of the requested information as possible, posting it here tomorrow morning. However I can already tell you that I am unable to find information about all the websites on the server. They all run independent of each other and the clients are able to upload their own websites and files.

Based upon that we can probably asume that there is old unsecured software running on at least a dozen webhotels. However the apache-installation runs with a lot of prohibited functions (see attached text-file), so my hope is, that it is enough to encounter insecure websites. Tell me if you need me to dig deeper into finding information about these websites.

The directory /var/www is owned by root and all subsequent directories are owned by the ftp-user. More information will follow tomorrow.

unSpawn 09-25-2012 01:30 PM

Quote:

Originally Posted by MortenOnDebian (Post 4789048)
I'll (..) gather (..) the requested information (..) information will follow tomorrow.

Since I asked you about 16 questions I know it will take time to process it. We'll just have to wait.

Noway2 09-28-2012 10:07 AM

MortenOnDebian, is there any update to this situation? Have you been able to obtain any of the requested data? Is the problem still occurring and have you been monitoring the situation?

MortenOnDebian 10-11-2012 06:45 AM

At last I have been able to obtain the requested information. I've uploaded it to a website of mine - see links below.

The server is running on VMware version 4.1 Update 1 - The newest update. It only runs the following services:
NTOP, ProFTPd, MySQL og Apache

The problem first occured around the 4th of September. After 3 days it started occuring once a day for a whole week.
Then it faded down to once every/every second week. The attacks normally occur in the late afternoon/evening + once
in the morning.

As an example one of the attacks occured the 24th of september from 5:42 pm until 6:05 pm when the power to the server
was cut off.

I was unable to only get the logwatch information from the above timespan, so the appended log is generated with
the "--range All" parameter. I've tried the command following commands with the only result of seing information about
the range parameter:

Code:

logwatch --detail High --service All --range '9/24/2012' --archives --numeric --save logwatch.log
Code:

logwatch --detail High --service All --range 'between 9/24/2012 17:00 and 9/24/2012 19:00' --archives --numeric --save logwatch.log
Files:
http://download.wep.dk/linux/logwatch.log
http://download.wep.dk/linux/rpmvfy.log
http://download.wep.dk/linux/logfile

Please ask if I have missed something. I hope I will be able to respond much faster.

Regards
Morten

unSpawn 10-11-2012 08:38 AM

Quote:

Originally Posted by MortenOnDebian (Post 4802900)
The server is running on VMware version 4.1 Update 1 - The newest update. It only runs the following services:
NTOP, ProFTPd, MySQL og Apache

...it also runs Xorg font server, Sendmail, RPC services and Webmin.
SELinux appears to be disabled.
Apache seems to be a standard configuration with all unnecessary modules loaded, mod_security seems to be in use.


Quote:

Originally Posted by MortenOnDebian (Post 4802900)
Please ask if I have missed something.

Please post:
- which access restrictions are in place,
- what hardening was performed (if any),
- and since you run Ntop, can you drill down to one period and show detailed stats by protocol, destination port, etc?


Quote:

Originally Posted by MortenOnDebian (Post 4802900)
I've uploaded it to a website of mine - see links below.

Thanks. Please obfuscate the servers canonical name and its IP address in the logwatch report and then use bzip2 to compress it. I'll then download and look at it. Due to the reboot (which you didn't mention earlier) process data may have less relevance but log data all the more.

MortenOnDebian 10-12-2012 09:12 AM

2 Attachment(s)
I also forgot to mention that IPTables isn't running. As far as I know there are no access restrictions. Futhermore there have not been done any hardening of the server.

The compressed logwatch file can now be found at:
http://download.wep.dk/linux/logwatch.log.bz2

Attached are to images from ntop showing the used protocols during the attacks. I've attached two images from 24th of september as well as from the 2nd of october, bot showing the same pattern.

unSpawn 10-12-2012 10:44 AM

Thanks. After a quick glance:

Code:

--------------------- ftpd-xferlog Begin ------------------------

 TOTAL KB OUT: 16809132KB  (16809MB)
 TOTAL KB IN: 101166367KB (101166MB)

Quite a lot uploaded that apparently is not downloaded via FTP...


Code:


 --------------------- httpd Begin ------------------------

        235 CD Images                (89122.01 MB),
    579328 Content pages            (19842.45 MB),
    545565 Images                  (18206.36 MB),
        502 Windows executable files (10578.18 MB),

Rearranging HTTP stats you see a clear division between what "regular" content is accessed and what you would not like to see as regular content. There are two accounts that upload ISO's (search for "avlinux") and the other one of them (search for "windows7_universal.iso") was reported back in 2010 for hosting a Chase Online bank phishing scam and I don't think he's an official "Photoshop CS5 Master Collection" reseller :-]

To link U/L / D/L's with said time frame of events happening 18(!) days ago grep your Proftp and Apache logs for the date of the 24th and grep for these two accounts. The output is probably better readable if you run it through a statistics reporter like webalizer.

MortenOnDebian 10-14-2012 11:38 AM

1 Attachment(s)
Hmm, I must admit I cannot get webalizer to display stats for the old log-files for September. I have exported them to a different server where I just installed webalizer, however it will only display data for October. As I said there had been an attack on the 2nd of October as well, so I went exploring this.

Even though the avlinux-domain has been logged for 17 gigabytes of data for the first 10 days of October (according to webalizer), there was no activity in the period of the attack. Same applies to the other domain you referred to. Going through the logs manually concluded this for both the 24th of September and the 2nd of October

However looking at the ntop graph I posted above, there seems to be a lot of torrent traffic. I have absolutely no clue from where this comes, but I guess it might have something to do with my problems. Do you have any suggestions?

Edit: Added the webalizer-graph for the "avlinux"-domain. As shown only a bit of the traffic occurs during the 2nd of October, where one of the attacks happened

unSpawn 10-15-2012 10:46 AM

The fundamental problems we hit are the limited amount of logging, running no firewall and less reporting than expected.

The latter ties in to a certain agree with the first but where I said
Quote:

Originally Posted by unSpawn (Post 4802989)
since you run Ntop, can you drill down to one period and show detailed stats by protocol, destination port, etc?

I did expect a detailed traffic report by source and destination hosts, ports, duration and transferred data size per incident. Ntop doesn't use OSI layer 7 classification AFAIK, instead it relies, similarly to nmap, on port mappings like /etc/services provide so you can't learn from that if traffic really is Bittorrent or not. Having a list of source and destination ports (listings scraped from its HTML reporting pages rather than graphs) means you can either correlate that with logging to you have or not. Having a list of source and destination hosts means you can see if there's certain hosts that cause this or if individual hosts traffic is low look for other causes.

Running without firewall means no protection from invalid traffic like bogons, wrong flags, wrong local service ports or traffic to remote services you should not allow, no rate limiting, no accounting of traffic and no logging. Since the incidents occur still w/o having found the cause we should work towards firewall rules that accomplish all of that. For starters please email me the output of
Code:

( ifconfig -a; route -n; iptables-save; netstat -antulpe ) > /tmp/log.txt
and I'll write the rules.

Finally I don't know if you're running any SAR like Atop, dstat or collectl (Mark: please don't ;-p) but if we can't get a fix on things then having resource utilization nfo could hold clues.

MortenOnDebian 10-15-2012 04:00 PM

When you talk about limited logging, you will for sure be glad when I tell you, that ntop doesn't display data about hosts and ports from before it was last started. So as the server has been forced to shut down, I cannot make ntop display any information about the source/destination of traffic during the attack.

All I can see is the amount of traffic. Maybe thats just some sort of misconfiguration from my side, but I cannot seem to find anything which changes this behaviour.

Quote:

( ifconfig -a; route -n; iptables-save; netstat -antulpe ) > /tmp/log.txt
As the iptables-save command was not found, I have just executed the command without that part. The output can be found at http://download.wep.dk/linux/log.txt


All times are GMT -5. The time now is 04:13 PM.