The company I work for received an e-mail saying that we are over our bandwidth. As everyone may know, when you house your servers in a Data Center they charge you for the bandwidth you use. Normally, our bill is right around $2,000 a month. The bill for the last two months came out to $20,000 for bandwidth!! Now, right off the back when they told us that I said we got hacked and use for DDOS attacks or possibly used as a reply of some sort.

I know some things for Linux but not as much as I should for this job which is sad. THAKN GOD for google! Are there any good places or pointers I can use to check on this issue before it gets worst. I think $20,000 is bad enough. I know I can check logs and stuff like that but I believe that I would have to do much more then that to check this. I would like to thank EVERYONE in advance for ANY help you can provide me here. THANKS!!

Somebody was definitely asleep at the wheel.

At a minimum, I'd demand to know just why they think you were "hacked". Demand as much detailed information as you can. You might wish to hire a consultant who can ask the right questions.

You might also consider retaining a lawyer.

But please - PLEASE - find another service that isn't going to pop any $20k surprises on you!

0 members found this post helpful.

Yea I hear that everyone was shocked! They did not even warn us that we was going over our limit or anything. NO TYPE OF NOTICE! But, I would love to find a way to figure what the heck our servers did to produce such large amounts of bandwidth!!

Ask your provider for a traffic report. The more detailed it is the better. Ask for a few samples of source and destination addresses and ports.


- What you must understand first is that thinking before acting is important, best start with reading the CERT Intruder Detection Checklist: http://web.archive.org/web/200801092...checklist.html and if your distribution has specific documentation wrt security read those too.
- When you're ready to act also understand that talking about things is not as efficient and does not enable us to help you as well as posting exact details.
- If you checked the logs already please post results. If you didn't then copy all logs, user auth databases, user shell history and crontabs over to a secure, clean workstation for processing and run 'logwatch' from there.


Do post:
- the distribution and release version,
- which services the machine or machines provide (including web-based management panels if any),
- which exact software versions and if the software was kept up to date,
- which logging, access restrictions is in place and hardening was performed,
- if there have been earlier breaches or anomalies,
- complete listings of running (output to a file in a temporary directory),
- any logwatch reporting done on the secure workstation (output to a file in a temporary directory),
- results from the actions performed as per the CERT Intruder Detection Checklist.

Please ask specific questions before performing if deemed necessary and please reply verbosely.

2 members found this post helpful.

Absolutely! But don't just "ask" - DEMAND.

And that's why I suggested taking somebody knowledgeable enough to ask the right questions - just to make sure they don't try to give you the runaround.

Yes, absolutely.

And that's why "reading the CERT intrusion checklist" is NOT the first thing you ought to be doing.

1. Figure out where the bandwidth has been going (traffic report).
2. Deal with your provider.
3. Deal with the outrageous bill.
4. Verify system integrity.
... and, way way down on the list ...
5. Read CERT literature

The whole "intrusion" thing is just a theory right now. Who knows - it might just be a clerical error on the part of your provider. Or it could just as easily be your application. For example, maybe each connection involves large amounts of data (meteorological data or high-end graphics, for instance). Or maybe it's an an RPC application, which, when busy, tends to generate a large number of short connections.

Deal with the FACTS first. The most pressing of which is your $20K bill

IMHO...

1 members found this post helpful.

As far as things go it would be prudent to start both approaches in tandem: if the provider needs convincing then knowing the system state and history is as it should be will support the OP's claim else if the system was abused (as say a warez D/L) then the OP will want to know. Leaving mitigation as somewhat as an afterthought might result in racking up an even higher bill while being swamped in protracted "taking somebody knowledgeable enough". FWIW also note that verifying system integrity may or may not reveal anything at all if changes were made outside the scope of what is to be verified, if no system integrity verification tools are available or if the OP doesn't know where to look and what to look for...

1 members found this post helpful.

Thank you for all the replies. I will proceed with the above tomorrow morning and reply with the information I get from it. Thank you everyone for your help!

What happened?

Sorry for not updating. There was a coding issue where the code was created to push out some PDF's and other documents and file types over to another server as a needed basis. The coder was someone who was let go and before he left, he re-wrote the code so that it does it ALL the time for EVERYTHING. It was gigs and gigs of stuff it was retarded the amount of things being sent out lol Well, I guess he wanted to leave his mark before he left. He did so with a $20,000 bill. Not sure how they going to handle on paying it or not.

Lol, a nice trick. Hope everything works out without paying the money.

Wow. Not to point out the insanely obvious, but your company needs to review its procedures on letting people go. You might want to have a quick code/system review for everything that person had access to, there may be other surprises.

Wow! That's jacked up! Time for a lawyer, I think. Record everything that happened and backup any data that supports that finding, then present it to a lawyer. Let the lawyer hunt him down and pay for what he did (literally or even figuratively, or both).

Yeah, with us, when we cut someone loose, they are escorted to their desk so they can pack their belongings and they are escorted out the door. I've even seen it where the person is immediately escorted out and someone brings his stuff to him in a box. About the only way they can get a heads-up and be able to leave a trap is if someone narcs for them.

Thanks for the update, keep 'em coming.

What sucks is that I'm the new guy here and BAM this happened. Well, hopefully from now on while I am here I will handle all this stuff so it won't happen again and hopefully they learned from this.