LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Security
User Name
Password
Linux - Security This forum is for all security related questions.
Questions, tips, system compromises, firewalls, etc. are all included here.

Notices

Reply
 
Search this Thread
Old 02-01-2012, 12:41 AM   #1
Nickalf
Member
 
Registered: Feb 2009
Location: on the InterNET most hours of my life
Distribution: Ubuntu Server v8.04 / v11.10 & Mac OsX (mainly for FCP & video/audio)
Posts: 35

Rep: Reputation: 15
Unhappy www-data account hijacked


My Ubuntu www-data system account has been hijacked from outside and is spewing SPAM at a rate of around 5,000 an hour..

My IP has now been blocked/blacklisted by Yahoo, MSN, AOL, Verizon, Baracuda etc etc. and my clients are getting p%^$&*ed as they can send their mail..

Before I go out of business, can anyone help (apart from me building a new server on a new IP - which I'm doing)

Thanks,
Nick...
 
Old 02-01-2012, 01:35 AM   #2
cbtshare
Member
 
Registered: Jul 2009
Posts: 561

Rep: Reputation: 42
Change your password.If use ssh to log into your servers change the password and also the port, if possible configure ssh key access.

check the /var/log/secure see what happened there.

I dont know what type of mail server you are using so just , research how to harden it.

Run scans. Tools like Lynis and Rootkit Hunter can give you alerts to possible security holes in your server. There are programs that maintain a hash or hash tree of all your bins and can alert you to changes.
 
Old 02-01-2012, 05:05 AM   #3
Nickalf
Member
 
Registered: Feb 2009
Location: on the InterNET most hours of my life
Distribution: Ubuntu Server v8.04 / v11.10 & Mac OsX (mainly for FCP & video/audio)
Posts: 35

Original Poster
Rep: Reputation: 15
Thanks for the comeback...

> Change your password.

I wish it were that simple - Tried that but they still come in/go out..

> check the /var/log/secure see what happened there.

There is no 'secure' file there ?

> I dont know what type of mail server you are using

Postfix/Dovecot..

> Tools like Lynis and Rootkit Hunter

I'll give them a try...

Thanks,
Nick...

P.s. Anyone else with suggestions ?
 
Old 02-01-2012, 11:19 AM   #4
cbtshare
Member
 
Registered: Jul 2009
Posts: 561

Rep: Reputation: 42
if they can come in and out install csf firewall and block their ipaddress with
Quote:
csf -d ipaddress
Make sure to configure server keys, to restrict access to ssh
 
Old 02-01-2012, 11:29 AM   #5
unSpawn
Moderator
 
Registered: May 2001
Posts: 26,988
Blog Entries: 54

Rep: Reputation: 2742Reputation: 2742Reputation: 2742Reputation: 2742Reputation: 2742Reputation: 2742Reputation: 2742Reputation: 2742Reputation: 2742Reputation: 2742Reputation: 2742
First of all please don't crosspost. It isn't the first time you've done this.
And ditch the ads, that's completely respectless towards this forum and its owner.


Quote:
Originally Posted by Nickalf View Post
My Ubuntu www-data system account has been hijacked from outside and is spewing SPAM at a rate of around 5,000 an hour..
If spam originates from your servers MTA then block outbound port TCP/25.
If spam originates from your web server (rogue Perl or PHP script or whatever else) stop the web service.

Before you decide a course of action it would be best if you provide:
- system details (OS, web stack software you run) and location (colocation, home, work),
- process information '( /bin/ps acxfwwwe 2>&1; /usr/sbin/lsof -Pwln 2>&1; /bin/ls -al /var/spool/cron 2>&1; /bin/netstat -anpe 2>&1; /usr/bin/lastlog 2>&1; /usr/bin/last 2>&1; /usr/bin/who -a 2>&1; /sbin/iptables -nvxL ) > /path/to/data.txt;'
- check your logs for any anomalies,
- check user homes and any directories any software in the web stack can write to for rogue files,
- list what you have done so far.

* BTW I would strongly suggest you don't delete any rogue files you find and definitely don't install any software yet.
 
Old 02-01-2012, 11:56 AM   #6
Nickalf
Member
 
Registered: Feb 2009
Location: on the InterNET most hours of my life
Distribution: Ubuntu Server v8.04 / v11.10 & Mac OsX (mainly for FCP & video/audio)
Posts: 35

Original Poster
Rep: Reputation: 15
Checking into the last two suggestions - thanks..
 
Old 02-01-2012, 05:03 PM   #7
Nickalf
Member
 
Registered: Feb 2009
Location: on the InterNET most hours of my life
Distribution: Ubuntu Server v8.04 / v11.10 & Mac OsX (mainly for FCP & video/audio)
Posts: 35

Original Poster
Rep: Reputation: 15
Arrow

cbtshare:
> if they can come in and out install csf firewall and block their ipaddress

I can't find where their IP address is - don't see it in any log

> Make sure to configure server keys, to restrict access to ssh

Where do I do that ?

unSpawn:
> If spam originates from your servers MTA

When I turn off Postfix, the SPAM builds in the queue showing it's 'From' www-data so it looks like it's internal to the outside world

> then block outbound port TCP/25.

What utility do I use for that ?

> If spam originates from your web server (rogue Perl or PHP script or whatever else) stop the web service.

Not sure if they got in through a rogue webapp or what.. Looks like it's been going on for awhile before it grew large enough for me to notice. Part of my business is to include client Website hosting, so stopping it is not an option (may have to build a 2nd one, but low to no budget)

Last edited by Nickalf; 02-01-2012 at 05:07 PM.
 
Old 02-01-2012, 08:48 PM   #8
unSpawn
Moderator
 
Registered: May 2001
Posts: 26,988
Blog Entries: 54

Rep: Reputation: 2742Reputation: 2742Reputation: 2742Reputation: 2742Reputation: 2742Reputation: 2742Reputation: 2742Reputation: 2742Reputation: 2742Reputation: 2742Reputation: 2742
Quote:
Originally Posted by Nickalf View Post
cbtshare:
> if they can come in and out install csf firewall and block their ipaddress
You already have iptables installed by default so you don't need to install LFD, CSF or whatever else frontend on offer.


Quote:
Originally Posted by Nickalf View Post
> Make sure to configure server keys, to restrict access to ssh
Where do I do that ?
Your distribution comes with documentation. I suggest you read it.
https://help.ubuntu.com/8.04/serverg...sh-server.html
https://help.ubuntu.com/community/SSH/OpenSSH/Keys


Quote:
Originally Posted by Nickalf View Post
unSpawn:
> If spam originates from your servers MTA
When I turn off Postfix, the SPAM builds in the queue showing it's 'From' www-data so it looks like it's internal to the outside world
Yes. The problem often is with software running in the web stack that has either problematic access permissions, home brewn software that doesn't properly escape user input, stale software versions or software that should not be accessible from the 'net at all. Although each case is unique with respect to web stack vulns your case is not unique to the ones we commonly handle in http://www.linuxquestions.org/questi...ux-security-4/.


Quote:
Originally Posted by Nickalf View Post
> then block outbound port TCP/25.
What utility do I use for that ?
iptables, see last question below.


Quote:
Originally Posted by Nickalf View Post
> If spam originates from your web server (rogue Perl or PHP script or whatever else) stop the web service.
Not sure if they got in through a rogue webapp or what.. Looks like it's been going on for awhile before it grew large enough for me to notice.
Start looking for PHP shells and assess installations of OsCommerce, Joomla, Drupal, gallery or other (3rd party?) plugins, any web-based management panels, statistics and shopping cart software that are older version that what is current. If anything found did not have updates released in the past year be suspicious.


Quote:
Originally Posted by Nickalf View Post
Part of my business is to include client Website hosting, so stopping it is not an option
A web site often is instrumental in creating and sustaining the right image for businesses. Anything that reflects badly on them they won't take kindly to. They trusted you and granted you (remember there's a gazillion budget hosts out there to choose from) the contract to host and take care of their business. In having your MTA send spam you do them a disservice. If they find out you don't take care of problems in the most definitve way, how would that reflect back on the trust they placed in you?
So IMHO right now your priority is to hang on to whatever shred of credibibility your business still has and yes, stopping indeed still is an option. You just have to look at the problem from the right point of view.

If you decide stopping the MTA is not possible then rate-limiting outbound TCP/25 via iptables or content-filtering email subjects via Postfix will also be of no help as they don't mitigate, stop or solve the problem. At least see if you can trace what executes email. Look for example at How To Log Emails Sent With PHP's mail() Function To Detect Form Spam and adapt it to your needs.


*BTW you did not answer everything or in as much detail as I would have liked to see. If you don't then I can only be of very limited help.
 
Old 02-02-2012, 01:36 AM   #9
colucix
Moderator
 
Registered: Sep 2003
Location: Bologna
Distribution: CentOS 6.5 OpenSuSE 12.3
Posts: 10,458

Rep: Reputation: 1941Reputation: 1941Reputation: 1941Reputation: 1941Reputation: 1941Reputation: 1941Reputation: 1941Reputation: 1941Reputation: 1941Reputation: 1941Reputation: 1941
Moved: This thread is more suitable in Linux - Security and has been moved accordingly to help your thread/question get the exposure it deserves.
 
Old 02-02-2012, 02:05 AM   #10
fukawi1
Member
 
Registered: Apr 2009
Location: Melbourne
Distribution: Fedora & CentOS
Posts: 854

Rep: Reputation: 189Reputation: 189
From a business perspective, I would remove your domain names from your forum profile, since they are shown on every post, and indexed by google. So any current or prospective client will be able to ascertain the level of service you provide based upon the questions you ask on the forum. Just sayin...

From a technical perspective, you should refer to your DR (Disaster Recovery) plan. Which as a provider of "professional" web hosting services, you will have. Which outlines EXACTLY what you should be doing right now.

Quote:
> then block outbound port TCP/25.

What utility do I use for that ?
I think this worries me the most, as it indicates to me that you have little to know understanding of firewalls, and therefore don't have one implemented. Frankly, its no bloody wonder you're server has been compromised, and your clients have every right to be dissatisfied with the service you are providing.
 
Old 02-02-2012, 03:23 AM   #11
Nickalf
Member
 
Registered: Feb 2009
Location: on the InterNET most hours of my life
Distribution: Ubuntu Server v8.04 / v11.10 & Mac OsX (mainly for FCP & video/audio)
Posts: 35

Original Poster
Rep: Reputation: 15
unSpawn:
Thanks for your helpful answers both here and at home..



fukawi1:
Thanks for your highly constructive answer - It really made my night.
 
Old 02-02-2012, 09:22 AM   #12
Noway2
Senior Member
 
Registered: Jul 2007
Distribution: Ubuntu 10.10, Slackware 64-current
Posts: 2,124

Rep: Reputation: 776Reputation: 776Reputation: 776Reputation: 776Reputation: 776Reputation: 776Reputation: 776
To block port 25, add a rule to your OUTPUT chain of iptables. As root:
Code:
 iptables -A OUTPUT -p tcp -m tcp --sport 25 -j drop
This will cause all outbound traffic on port 25 (SMTP) to be blocked. You will loose the ability to send email.

Second, you will need to perform an investigation into the source. unSpawn has provided the list of initial things to check. Based upon the information provided, chiefly that the origin of the email appears to be www-data, you will need to determine which web "host" or service is responsible. Do you have a date whence the problem began? If so, can you locate any files that have been modified on or around that date? Otherwise, you will need to isolate the sources. I would recommend shutting down all of the hosted pages / sites, clear the email queue (Postfix has a flush command that will do this), verify that nothing is happening. Then start re-enabling the sites. You can choose whatever method you feel is appropraite, one at a time, a group, or do a binary search splitting the difference. The objective is to nail down which one(s) is (are) responsible.

Also, how up to date is your system? One thing about Ubuntu is that it is very active in prompting you for application and security updates.

Edit: fukawi1's post was rather blunt, and while it could have been phrased better, the point is valid. As you have accepted the responsibility for administering the system, you have the obligation to familiarize yourself with the basic tools necessary to perform this function. Iptables is one of these tools and is an interface to the firewall built into the Linux Kernel. We can address your learning needs and make some recommendations for ways for you to get started, but lets focus on getting your server out of the weeds first.

Last edited by Noway2; 02-02-2012 at 09:25 AM.
 
Old 02-02-2012, 10:21 AM   #13
unSpawn
Moderator
 
Registered: May 2001
Posts: 26,988
Blog Entries: 54

Rep: Reputation: 2742Reputation: 2742Reputation: 2742Reputation: 2742Reputation: 2742Reputation: 2742Reputation: 2742Reputation: 2742Reputation: 2742Reputation: 2742Reputation: 2742
* Additionally you have marked this thread "[SOLVED]" while IMHO it is not. I haven't seen spam origin and the fix posted. If you fixed it already please reciprocate by posting your solution, TIA.
 
Old 02-07-2012, 03:11 AM   #14
Nickalf
Member
 
Registered: Feb 2009
Location: on the InterNET most hours of my life
Distribution: Ubuntu Server v8.04 / v11.10 & Mac OsX (mainly for FCP & video/audio)
Posts: 35

Original Poster
Rep: Reputation: 15
As so much damage was done (IP blocked by many of the big mail hosts) and needing a faster solution to help my clients, apart from being talked down to (Newbies have feelings too)- I [SOLVED] it by quickly building a new server, transferring everything over and shutting down the old one..

Thanks for your guidance...
 
Old 02-07-2012, 05:05 AM   #15
Noway2
Senior Member
 
Registered: Jul 2007
Distribution: Ubuntu 10.10, Slackware 64-current
Posts: 2,124

Rep: Reputation: 776Reputation: 776Reputation: 776Reputation: 776Reputation: 776Reputation: 776Reputation: 776
Nickalf, unless you have identified how your server was compromised and taken steps to correct the problem, it is likely that they will return and re-infect your new server. This is one of the reasons why we actively discourage the wipe and re-install methodology. What's more is that if you go to the effort to get yourself de-listed from the RBLs and continue to have problems you will find getting cleared again to be more difficult.

Do you still have the old server or at least an image of it, along with the log and web files? If so, you may still be able to determine the root cause of the "infestation".
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
www-data account hijacked Nickalf Linux - Server 1 02-02-2012 01:35 AM
www mail account manager sloniupl Linux - Newbie 2 03-07-2011 03:39 AM
www-data tommytomato Linux - Newbie 35 03-19-2010 02:03 AM
Hijacked email account agentchange Suse/Novell 7 07-08-2006 04:18 PM
add www-data user restless Linux - Newbie 1 06-01-2004 07:51 AM


All times are GMT -5. The time now is 03:29 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration