LinuxQuestions.org
Go Job Hunting at the LQ Job Marketplace
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Security
User Name
Password
Linux - Security This forum is for all security related questions.
Questions, tips, system compromises, firewalls, etc. are all included here.

Notices



Reply
 
Search this Thread
Old 09-25-2012, 12:04 PM   #1
MortenOnDebian
LQ Newbie
 
Registered: Aug 2010
Posts: 14

Rep: Reputation: 1
Server suddently uploads huge amounts of data


Hello,

I'm having a problem with a Centos server. Sometimes it is almost like it is getting attacked or maybe more correctly attacking others. While this happens the network is completely overfloded and jammed.

If you look at the attached image you can see that my throughput for the server suddently rises enormously. The server is running a bit under 1000 websites as well as ftp for those. A few other services run in the background too. However I am not able to identify the source of my problems nor where the data is sent to.

Ntop is configured only to listen to traffic intended for the server itself (Non Promiscuous Mode).

I cannot seem to find anything in the logs, but that is probably because I don't know what to look for. Anybody got an idea for how to identify the problem?

Best regards
Morten
Attached Images
File Type: png throughput.PNG (190.7 KB, 41 views)

Last edited by MortenOnDebian; 09-25-2012 at 12:07 PM.
 
Old 09-25-2012, 12:09 PM   #2
2armz
Member
 
Registered: Sep 2012
Location: Garner NC
Distribution: Fedora 17
Posts: 35

Rep: Reputation: Disabled
Change Root Password, make sure you can't ssh into the machine as root. Also if I can remember correctly "check" the httpd.conf. See if there are any changes to it, like a share directory. I could be wrong, but sounds like someone has hacked you.

Last edited by 2armz; 09-25-2012 at 01:09 PM.
 
Old 09-25-2012, 12:44 PM   #3
unSpawn
Moderator
 
Registered: May 2001
Posts: 27,791
Blog Entries: 54

Rep: Reputation: 2980Reputation: 2980Reputation: 2980Reputation: 2980Reputation: 2980Reputation: 2980Reputation: 2980Reputation: 2980Reputation: 2980Reputation: 2980Reputation: 2980
First of all please disregard any "advice" suggesting you should edit, install, remove or change anything on the machine as that could hamper analysis.


Quote:
Originally Posted by MortenOnDebian View Post
I'm having a problem with a Centos server. (..)
The server is running a bit under 1000 websites as well as ftp for those. (..)
A few other services run in the background too. (..)
0. Please post:
- the release version, if it runs (on) any form of virtualization (VMware, XEN, etc), process partitioning (LXC, OpenVZ, etc),
- which services the machine itself provides including web-based management panels, statistics, web log, forum, shopping cart, plugins and other software, and including what the web sites run, their exact software versions and if that software was kept up to date,
- which logging, access restrictions is in place and what hardening was performed,
- if there have been earlier breaches or anomalies,
- since when and how regularly this problem manifests itself, and
- anything else tangible, factual and supported by data worth mentioning.

1. Please attach as plain text or post, preferably in [code]vBB tags[/code] the complete listings of these commands (preferably first cd to /dev/shm or else if you must /tmp):
Code:
( /bin/ps axfwwwe -opid,ppid,uid,context,cmd 2>&1; /usr/sbin/lsof -Pwln 2>&1; \
find /var/spool/cron 2>&1; /bin/netstat -anpe 2>&1; /usr/bin/lastlog 2>&1; /usr/bin/last \
-wai 2>&1; /usr/bin/who -a 2>&1 ) > logfile 2>&1
Code:
/bin/rpm  --nodeps --noscripts --notriggers -Vva 2>&1|/bin/grep -v "\.\{8\}" > rpmvfy.log 2>&1
Running all system and daemon logs through Logwatch with the
Code:
--detail High --service All --range All --archives --numeric --save logwatch.log
args. The latter is best done by copying logs to a separate machine or workstation and running Logwatch from there.

2. Since you run Ntop, can you drill down to one period and show detailed stats by protocol, destination port, etc?

Also please ask specific questions before performing if necessary, please reply verbosely and do stay with the thread (subscribe?) until completion and reply as soon as possible when replies are posted.

Last edited by unSpawn; 09-25-2012 at 12:53 PM. Reason: //More *is* more
 
Old 09-25-2012, 01:02 PM   #4
2armz
Member
 
Registered: Sep 2012
Location: Garner NC
Distribution: Fedora 17
Posts: 35

Rep: Reputation: Disabled
Sorry I didn't aim to hamper anything, just was a thought. Just incase someone has gotten into the system, and changed someone changed something. I worded the sentence wrong there. Not my strongest skill.
 
Old 09-25-2012, 01:19 PM   #5
unSpawn
Moderator
 
Registered: May 2001
Posts: 27,791
Blog Entries: 54

Rep: Reputation: 2980Reputation: 2980Reputation: 2980Reputation: 2980Reputation: 2980Reputation: 2980Reputation: 2980Reputation: 2980Reputation: 2980Reputation: 2980Reputation: 2980
Quote:
Originally Posted by 2armz View Post
skill.
Since you're quite new to LQ and since you did mention the word, just so you know the LQ Linux Security forum aims to be as fast and efficient as possible especially when faced with potential breaches of security. Those who practice Incident Response / sniper forensics here like to keep the SNR as low as possible and the information presented to be relevant, factual and complete. Those w/o IR practical knowledge, those who won't stay with a thread until completion and those who think about posting a "don't worry", "I think" or "I guess" type of one-liner best wait until one of the handlers arrives and if they can't resist the compulsion to post just point the OP to the (outdated but still useful) CERT Intruder Detection Checklist,
TIA.
 
Old 09-25-2012, 01:59 PM   #6
MortenOnDebian
LQ Newbie
 
Registered: Aug 2010
Posts: 14

Original Poster
Rep: Reputation: 1
I'll try to gather as much of the requested information as possible, posting it here tomorrow morning. However I can already tell you that I am unable to find information about all the websites on the server. They all run independent of each other and the clients are able to upload their own websites and files.

Based upon that we can probably asume that there is old unsecured software running on at least a dozen webhotels. However the apache-installation runs with a lot of prohibited functions (see attached text-file), so my hope is, that it is enough to encounter insecure websites. Tell me if you need me to dig deeper into finding information about these websites.

The directory /var/www is owned by root and all subsequent directories are owned by the ftp-user. More information will follow tomorrow.
Attached Files
File Type: txt disabled_functions.txt (278 Bytes, 17 views)
 
Old 09-25-2012, 02:30 PM   #7
unSpawn
Moderator
 
Registered: May 2001
Posts: 27,791
Blog Entries: 54

Rep: Reputation: 2980Reputation: 2980Reputation: 2980Reputation: 2980Reputation: 2980Reputation: 2980Reputation: 2980Reputation: 2980Reputation: 2980Reputation: 2980Reputation: 2980
Quote:
Originally Posted by MortenOnDebian View Post
I'll (..) gather (..) the requested information (..) information will follow tomorrow.
Since I asked you about 16 questions I know it will take time to process it. We'll just have to wait.
 
Old 09-28-2012, 11:07 AM   #8
Noway2
Senior Member
 
Registered: Jul 2007
Distribution: Ubuntu 10.10, Slackware 64-current
Posts: 2,124

Rep: Reputation: 776Reputation: 776Reputation: 776Reputation: 776Reputation: 776Reputation: 776Reputation: 776
MortenOnDebian, is there any update to this situation? Have you been able to obtain any of the requested data? Is the problem still occurring and have you been monitoring the situation?
 
Old 10-11-2012, 07:45 AM   #9
MortenOnDebian
LQ Newbie
 
Registered: Aug 2010
Posts: 14

Original Poster
Rep: Reputation: 1
At last I have been able to obtain the requested information. I've uploaded it to a website of mine - see links below.

The server is running on VMware version 4.1 Update 1 - The newest update. It only runs the following services:
NTOP, ProFTPd, MySQL og Apache

The problem first occured around the 4th of September. After 3 days it started occuring once a day for a whole week.
Then it faded down to once every/every second week. The attacks normally occur in the late afternoon/evening + once
in the morning.

As an example one of the attacks occured the 24th of september from 5:42 pm until 6:05 pm when the power to the server
was cut off.

I was unable to only get the logwatch information from the above timespan, so the appended log is generated with
the "--range All" parameter. I've tried the command following commands with the only result of seing information about
the range parameter:

Code:
logwatch --detail High --service All --range '9/24/2012' --archives --numeric --save logwatch.log
Code:
logwatch --detail High --service All --range 'between 9/24/2012 17:00 and 9/24/2012 19:00' --archives --numeric --save logwatch.log
Files:
http://download.wep.dk/linux/logwatch.log
http://download.wep.dk/linux/rpmvfy.log
http://download.wep.dk/linux/logfile

Please ask if I have missed something. I hope I will be able to respond much faster.

Regards
Morten
 
Old 10-11-2012, 09:38 AM   #10
unSpawn
Moderator
 
Registered: May 2001
Posts: 27,791
Blog Entries: 54

Rep: Reputation: 2980Reputation: 2980Reputation: 2980Reputation: 2980Reputation: 2980Reputation: 2980Reputation: 2980Reputation: 2980Reputation: 2980Reputation: 2980Reputation: 2980
Quote:
Originally Posted by MortenOnDebian View Post
The server is running on VMware version 4.1 Update 1 - The newest update. It only runs the following services:
NTOP, ProFTPd, MySQL og Apache
...it also runs Xorg font server, Sendmail, RPC services and Webmin.
SELinux appears to be disabled.
Apache seems to be a standard configuration with all unnecessary modules loaded, mod_security seems to be in use.


Quote:
Originally Posted by MortenOnDebian View Post
Please ask if I have missed something.
Please post:
- which access restrictions are in place,
- what hardening was performed (if any),
- and since you run Ntop, can you drill down to one period and show detailed stats by protocol, destination port, etc?


Quote:
Originally Posted by MortenOnDebian View Post
I've uploaded it to a website of mine - see links below.
Thanks. Please obfuscate the servers canonical name and its IP address in the logwatch report and then use bzip2 to compress it. I'll then download and look at it. Due to the reboot (which you didn't mention earlier) process data may have less relevance but log data all the more.
 
Old 10-12-2012, 10:12 AM   #11
MortenOnDebian
LQ Newbie
 
Registered: Aug 2010
Posts: 14

Original Poster
Rep: Reputation: 1
I also forgot to mention that IPTables isn't running. As far as I know there are no access restrictions. Futhermore there have not been done any hardening of the server.

The compressed logwatch file can now be found at:
http://download.wep.dk/linux/logwatch.log.bz2

Attached are to images from ntop showing the used protocols during the attacks. I've attached two images from 24th of september as well as from the 2nd of october, bot showing the same pattern.
Attached Images
File Type: png 24 september.PNG (225.2 KB, 15 views)
File Type: png 2 oct.PNG (33.5 KB, 12 views)
 
Old 10-12-2012, 11:44 AM   #12
unSpawn
Moderator
 
Registered: May 2001
Posts: 27,791
Blog Entries: 54

Rep: Reputation: 2980Reputation: 2980Reputation: 2980Reputation: 2980Reputation: 2980Reputation: 2980Reputation: 2980Reputation: 2980Reputation: 2980Reputation: 2980Reputation: 2980
Thanks. After a quick glance:

Code:
 --------------------- ftpd-xferlog Begin ------------------------ 

 TOTAL KB OUT: 16809132KB  (16809MB)
 TOTAL KB IN: 101166367KB (101166MB)
Quite a lot uploaded that apparently is not downloaded via FTP...


Code:
 
 --------------------- httpd Begin ------------------------ 

        235 CD Images                (89122.01 MB),
     579328 Content pages            (19842.45 MB),
     545565 Images                   (18206.36 MB),
        502 Windows executable files (10578.18 MB),
Rearranging HTTP stats you see a clear division between what "regular" content is accessed and what you would not like to see as regular content. There are two accounts that upload ISO's (search for "avlinux") and the other one of them (search for "windows7_universal.iso") was reported back in 2010 for hosting a Chase Online bank phishing scam and I don't think he's an official "Photoshop CS5 Master Collection" reseller :-]

To link U/L / D/L's with said time frame of events happening 18(!) days ago grep your Proftp and Apache logs for the date of the 24th and grep for these two accounts. The output is probably better readable if you run it through a statistics reporter like webalizer.
 
Old 10-14-2012, 12:38 PM   #13
MortenOnDebian
LQ Newbie
 
Registered: Aug 2010
Posts: 14

Original Poster
Rep: Reputation: 1
Hmm, I must admit I cannot get webalizer to display stats for the old log-files for September. I have exported them to a different server where I just installed webalizer, however it will only display data for October. As I said there had been an attack on the 2nd of October as well, so I went exploring this.

Even though the avlinux-domain has been logged for 17 gigabytes of data for the first 10 days of October (according to webalizer), there was no activity in the period of the attack. Same applies to the other domain you referred to. Going through the logs manually concluded this for both the 24th of September and the 2nd of October

However looking at the ntop graph I posted above, there seems to be a lot of torrent traffic. I have absolutely no clue from where this comes, but I guess it might have something to do with my problems. Do you have any suggestions?

Edit: Added the webalizer-graph for the "avlinux"-domain. As shown only a bit of the traffic occurs during the 2nd of October, where one of the attacks happened
Attached Images
File Type: png daily october.PNG (12.9 KB, 14 views)

Last edited by MortenOnDebian; 10-14-2012 at 12:42 PM.
 
Old 10-15-2012, 11:46 AM   #14
unSpawn
Moderator
 
Registered: May 2001
Posts: 27,791
Blog Entries: 54

Rep: Reputation: 2980Reputation: 2980Reputation: 2980Reputation: 2980Reputation: 2980Reputation: 2980Reputation: 2980Reputation: 2980Reputation: 2980Reputation: 2980Reputation: 2980
The fundamental problems we hit are the limited amount of logging, running no firewall and less reporting than expected.

The latter ties in to a certain agree with the first but where I said
Quote:
Originally Posted by unSpawn View Post
since you run Ntop, can you drill down to one period and show detailed stats by protocol, destination port, etc?
I did expect a detailed traffic report by source and destination hosts, ports, duration and transferred data size per incident. Ntop doesn't use OSI layer 7 classification AFAIK, instead it relies, similarly to nmap, on port mappings like /etc/services provide so you can't learn from that if traffic really is Bittorrent or not. Having a list of source and destination ports (listings scraped from its HTML reporting pages rather than graphs) means you can either correlate that with logging to you have or not. Having a list of source and destination hosts means you can see if there's certain hosts that cause this or if individual hosts traffic is low look for other causes.

Running without firewall means no protection from invalid traffic like bogons, wrong flags, wrong local service ports or traffic to remote services you should not allow, no rate limiting, no accounting of traffic and no logging. Since the incidents occur still w/o having found the cause we should work towards firewall rules that accomplish all of that. For starters please email me the output of
Code:
( ifconfig -a; route -n; iptables-save; netstat -antulpe ) > /tmp/log.txt
and I'll write the rules.

Finally I don't know if you're running any SAR like Atop, dstat or collectl (Mark: please don't ;-p) but if we can't get a fix on things then having resource utilization nfo could hold clues.
 
Old 10-15-2012, 05:00 PM   #15
MortenOnDebian
LQ Newbie
 
Registered: Aug 2010
Posts: 14

Original Poster
Rep: Reputation: 1
When you talk about limited logging, you will for sure be glad when I tell you, that ntop doesn't display data about hosts and ports from before it was last started. So as the server has been forced to shut down, I cannot make ntop display any information about the source/destination of traffic during the attack.

All I can see is the amount of traffic. Maybe thats just some sort of misconfiguration from my side, but I cannot seem to find anything which changes this behaviour.

Quote:
( ifconfig -a; route -n; iptables-save; netstat -antulpe ) > /tmp/log.txt
As the iptables-save command was not found, I have just executed the command without that part. The output can be found at http://download.wep.dk/linux/log.txt
 
  


Reply

Tags
hacking, log, network monitoring, ntop


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] MySQL backup - how to deal with large amounts of data? karll Linux - Server 8 02-18-2011 10:51 AM
Best utility to copy massive amounts of data king0770 Linux - Hardware 2 07-16-2010 02:45 PM
Searching through massive amounts of Data sxa Linux - Software 5 02-27-2009 10:42 PM
how to limit swapping - prevent processes allocating huge amounts of memory david@linuxquestions Linux - General 10 12-21-2006 08:26 AM
rm command is choking on large amounts of data? Jello Linux - General 18 02-28-2003 08:11 PM


All times are GMT -5. The time now is 11:55 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration