LinuxQuestions.org
Support LQ: Use code LQ3 and save $3 on Domain Registration
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Security
User Name
Password
Linux - Security This forum is for all security related questions.
Questions, tips, system compromises, firewalls, etc. are all included here.

Notices

Reply
 
Search this Thread
Old 04-19-2011, 10:13 PM   #1
TheNewGuy2936
LQ Newbie
 
Registered: Apr 2011
Location: Brooklyn NYC
Distribution: Redhat & CentOS
Posts: 23

Rep: Reputation: 6
Excessive band-width usage = Major Problem!


The company I work for received an e-mail saying that we are over our bandwidth. As everyone may know, when you house your servers in a Data Center they charge you for the bandwidth you use. Normally, our bill is right around $2,000 a month. The bill for the last two months came out to $20,000 for bandwidth!! Now, right off the back when they told us that I said we got hacked and use for DDOS attacks or possibly used as a reply of some sort.

I know some things for Linux but not as much as I should for this job which is sad. THAKN GOD for google! Are there any good places or pointers I can use to check on this issue before it gets worst. I think $20,000 is bad enough. I know I can check logs and stuff like that but I believe that I would have to do much more then that to check this. I would like to thank EVERYONE in advance for ANY help you can provide me here. THANKS!!
 
Click here to see the post LQ members have rated as the most helpful post in this thread.
Old 04-19-2011, 11:01 PM   #2
paulsm4
Guru
 
Registered: Mar 2004
Distribution: SusE 8.2
Posts: 5,863
Blog Entries: 1

Rep: Reputation: Disabled
Somebody was definitely asleep at the wheel.

At a minimum, I'd demand to know just why they think you were "hacked". Demand as much detailed information as you can. You might wish to hire a consultant who can ask the right questions.

You might also consider retaining a lawyer.

But please - PLEASE - find another service that isn't going to pop any $20k surprises on you!
 
0 members found this post helpful.
Old 04-19-2011, 11:11 PM   #3
TheNewGuy2936
LQ Newbie
 
Registered: Apr 2011
Location: Brooklyn NYC
Distribution: Redhat & CentOS
Posts: 23

Original Poster
Rep: Reputation: 6
Yea I hear that everyone was shocked! They did not even warn us that we was going over our limit or anything. NO TYPE OF NOTICE! But, I would love to find a way to figure what the heck our servers did to produce such large amounts of bandwidth!!
 
Old 04-20-2011, 01:01 AM   #4
unSpawn
Moderator
 
Registered: May 2001
Posts: 27,304
Blog Entries: 54

Rep: Reputation: 2855Reputation: 2855Reputation: 2855Reputation: 2855Reputation: 2855Reputation: 2855Reputation: 2855Reputation: 2855Reputation: 2855Reputation: 2855Reputation: 2855
Quote:
Originally Posted by TheNewGuy2936 View Post
The company I work for received an e-mail saying that we are over our bandwidth. (..) The bill for the last two months came out to $20,000 for bandwidth
Ask your provider for a traffic report. The more detailed it is the better. Ask for a few samples of source and destination addresses and ports.


Quote:
Originally Posted by TheNewGuy2936 View Post
I know some things for Linux but not as much as I should for this job (..) I know I can check logs and stuff like that
- What you must understand first is that thinking before acting is important, best start with reading the CERT Intruder Detection Checklist: http://web.archive.org/web/200801092...checklist.html and if your distribution has specific documentation wrt security read those too.
- When you're ready to act also understand that talking about things is not as efficient and does not enable us to help you as well as posting exact details.
- If you checked the logs already please post results. If you didn't then copy all logs, user auth databases, user shell history and crontabs over to a secure, clean workstation for processing and run 'logwatch' from there.


Quote:
Originally Posted by TheNewGuy2936 View Post
but I believe that I would have to do much more then that to check this.
Do post:
- the distribution and release version,
- which services the machine or machines provide (including web-based management panels if any),
- which exact software versions and if the software was kept up to date,
- which logging, access restrictions is in place and hardening was performed,
- if there have been earlier breaches or anomalies,
- complete listings of running
Code:
( ps axfwwwe 2>&1; netstat -anpe 2>&1; lsof -Pwln 2>&1; who 2>&1; last 2>&1; rpm -Vva 2>&1|grep -v "^\.\{8\}" )
(output to a file in a temporary directory),
- any logwatch reporting done on the secure workstation
Code:
logwatch --numeric --detail 5 --service all --range All --archives --print 2>&1;
(output to a file in a temporary directory),
- results from the actions performed as per the CERT Intruder Detection Checklist.

Please ask specific questions before performing if deemed necessary and please reply verbosely.
 
2 members found this post helpful.
Old 04-20-2011, 09:39 AM   #5
paulsm4
Guru
 
Registered: Mar 2004
Distribution: SusE 8.2
Posts: 5,863
Blog Entries: 1

Rep: Reputation: Disabled
Quote:
Ask your provider for a traffic report.
Absolutely! But don't just "ask" - DEMAND.

And that's why I suggested taking somebody knowledgeable enough to ask the right questions - just to make sure they don't try to give you the runaround.

Quote:
What you must understand first is that thinking before acting is important.
Yes, absolutely.

And that's why "reading the CERT intrusion checklist" is NOT the first thing you ought to be doing.

1. Figure out where the bandwidth has been going (traffic report).
2. Deal with your provider.
3. Deal with the outrageous bill.
4. Verify system integrity.
... and, way way down on the list ...
5. Read CERT literature

The whole "intrusion" thing is just a theory right now. Who knows - it might just be a clerical error on the part of your provider. Or it could just as easily be your application. For example, maybe each connection involves large amounts of data (meteorological data or high-end graphics, for instance). Or maybe it's an an RPC application, which, when busy, tends to generate a large number of short connections.

Deal with the FACTS first. The most pressing of which is your $20K bill

IMHO...

Last edited by paulsm4; 04-20-2011 at 10:35 AM.
 
1 members found this post helpful.
Old 04-20-2011, 04:55 PM   #6
unSpawn
Moderator
 
Registered: May 2001
Posts: 27,304
Blog Entries: 54

Rep: Reputation: 2855Reputation: 2855Reputation: 2855Reputation: 2855Reputation: 2855Reputation: 2855Reputation: 2855Reputation: 2855Reputation: 2855Reputation: 2855Reputation: 2855
As far as things go it would be prudent to start both approaches in tandem: if the provider needs convincing then knowing the system state and history is as it should be will support the OP's claim else if the system was abused (as say a warez D/L) then the OP will want to know. Leaving mitigation as somewhat as an afterthought might result in racking up an even higher bill while being swamped in protracted "taking somebody knowledgeable enough". FWIW also note that verifying system integrity may or may not reveal anything at all if changes were made outside the scope of what is to be verified, if no system integrity verification tools are available or if the OP doesn't know where to look and what to look for...
 
1 members found this post helpful.
Old 04-21-2011, 12:23 AM   #7
TheNewGuy2936
LQ Newbie
 
Registered: Apr 2011
Location: Brooklyn NYC
Distribution: Redhat & CentOS
Posts: 23

Original Poster
Rep: Reputation: 6
Thank you for all the replies. I will proceed with the above tomorrow morning and reply with the information I get from it. Thank you everyone for your help!
 
Old 04-25-2011, 08:09 AM   #8
archtoad6
Senior Member
 
Registered: Oct 2004
Location: Houston, TX (usa)
Distribution: MEPIS, Debian, Knoppix,
Posts: 4,727
Blog Entries: 15

Rep: Reputation: 231Reputation: 231Reputation: 231
What happened?
 
Old 04-26-2011, 10:14 AM   #9
TheNewGuy2936
LQ Newbie
 
Registered: Apr 2011
Location: Brooklyn NYC
Distribution: Redhat & CentOS
Posts: 23

Original Poster
Rep: Reputation: 6
Sorry for not updating. There was a coding issue where the code was created to push out some PDF's and other documents and file types over to another server as a needed basis. The coder was someone who was let go and before he left, he re-wrote the code so that it does it ALL the time for EVERYTHING. It was gigs and gigs of stuff it was retarded the amount of things being sent out lol Well, I guess he wanted to leave his mark before he left. He did so with a $20,000 bill. Not sure how they going to handle on paying it or not.
 
Old 04-26-2011, 10:34 AM   #10
x64
LQ Newbie
 
Registered: Apr 2011
Location: Berlin
Distribution: Mint x64 Gnome
Posts: 7

Rep: Reputation: 0
Lol, a nice trick. Hope everything works out without paying the money.
 
Old 04-26-2011, 11:14 AM   #11
Hangdog42
LQ Veteran
 
Registered: Feb 2003
Location: Maryland
Distribution: Slackware
Posts: 7,785
Blog Entries: 1

Rep: Reputation: 414Reputation: 414Reputation: 414Reputation: 414Reputation: 414
Quote:
Originally Posted by TheNewGuy2936
The coder was someone who was let go and before he left, he re-wrote the code so that it does it ALL the time for EVERYTHING.
Wow. Not to point out the insanely obvious, but your company needs to review its procedures on letting people go. You might want to have a quick code/system review for everything that person had access to, there may be other surprises.
 
Old 04-26-2011, 11:17 AM   #12
unixfool
Member
 
Registered: May 2005
Location: Northern VA
Distribution: Slackware, Ubuntu, FreeBSD, OpenBSD, OS X
Posts: 781
Blog Entries: 8

Rep: Reputation: 157Reputation: 157
Wow! That's jacked up! Time for a lawyer, I think. Record everything that happened and backup any data that supports that finding, then present it to a lawyer. Let the lawyer hunt him down and pay for what he did (literally or even figuratively, or both).
 
Old 04-26-2011, 11:20 AM   #13
unixfool
Member
 
Registered: May 2005
Location: Northern VA
Distribution: Slackware, Ubuntu, FreeBSD, OpenBSD, OS X
Posts: 781
Blog Entries: 8

Rep: Reputation: 157Reputation: 157
Quote:
Originally Posted by Hangdog42 View Post
Wow. Not to point out the insanely obvious, but your company needs to review its procedures on letting people go. You might want to have a quick code/system review for everything that person had access to, there may be other surprises.
Yeah, with us, when we cut someone loose, they are escorted to their desk so they can pack their belongings and they are escorted out the door. I've even seen it where the person is immediately escorted out and someone brings his stuff to him in a box. About the only way they can get a heads-up and be able to leave a trap is if someone narcs for them.
 
Old 04-27-2011, 04:43 AM   #14
archtoad6
Senior Member
 
Registered: Oct 2004
Location: Houston, TX (usa)
Distribution: MEPIS, Debian, Knoppix,
Posts: 4,727
Blog Entries: 15

Rep: Reputation: 231Reputation: 231Reputation: 231
Thanks for the update, keep 'em coming.
 
Old 04-27-2011, 08:23 AM   #15
TheNewGuy2936
LQ Newbie
 
Registered: Apr 2011
Location: Brooklyn NYC
Distribution: Redhat & CentOS
Posts: 23

Original Poster
Rep: Reputation: 6
What sucks is that I'm the new guy here and BAM this happened. Well, hopefully from now on while I am here I will handle all this stuff so it won't happen again and hopefully they learned from this.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Excessive CPU Usage Smarajit Linux - Newbie 2 04-11-2007 11:02 AM
Monitor band width usages for each users in LAN sundar1712 Red Hat 2 07-13-2006 06:11 PM
Controlling the band width…. shahg_shahg Linux - Networking 1 12-20-2004 02:31 PM
Band width graphics tommytomato Linux - General 2 09-23-2004 09:28 AM
Dividing band width. Ciccio Linux - Networking 2 06-20-2003 12:30 PM


All times are GMT -5. The time now is 01:39 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration