I manage a dedicated webserver running OpenSuse 11 which is currently hosting about 30 sites.
I have never had any big problems until these last 2-3 months.
One site after the other was being hacked and the unwanted visitor installed all kind of php shell scripts followed by torrent servers, ... etc.

All hacked sites were sites using Joomla, so what I did was to close down those sites one by one.
Well, I guess we all know Joomla is not a great solution if you just install it out of the box like those users were doing.

When trying to trace the intruder only some african junk IPs and IPs from a company selling VPN connections thru paypal show up (yeah great, love those guys ... do they really think that serious VPN users will pay with paypal)

I checked all apache and FTP logs (yes, he even managed to get some FTP login) but only those damn 'proxy' IPs come up.

The weird thing is that the guy seems to know how the server was 'build' since he manages to copy stuff from one site to the other. That is why I am suspecting someone who worked for a clients company, but I need proof.

One way would be to let him hack a site and try to feed him something that would make him traceable, but what?

Any idea is welcome

PS : such a experience surely gives you some training on how to use grep

While I do understand you crave vengeance and with all due respect but it is highly unlikely the case possesses aspect that law enforcement would be interested in in terms of investigation let alone prosecution. I strongly suggest you do not waste time tracing the perpetrator and concentrate on fixing things proper.

Agree with unSpawn. Fix the current issue is the most important thing. It seems your problem is with Joomla, I think you'd better go to Joomla website to see whether there's any security update or not.

The chances are that the attacker is working from another compromised server anyway.

These guys often hop between several servers - so it would be nigh on impossible to trace.

As others have said - concentrate your efforts on fixing the weaknesses in your hosting environment (Joomla).

Also ensure that your php installation is secure (if you haven't already done so).

tracking website hackers or trapping them are really Impossible. When you are started using joomla you should have subscribed for the alert mails about patch/bug fix. Even after you are hacked pl report it to Joomla.

How do you know that the Website is hacked through Joomla? did you got any footprint or any proof? If you have please share with us.

If the hacker planted some shell scripting mean!!!, there is some other thing left un-covered from the Operating System side.

reg

The problem is that it would be somehow foolish for me to fix the Joomla issue because those sites are made by another guy who claims making good sites for low cost while I ask ask good bucks myself (I work on a freelance base for that company).
So if I fix 'his' issues I would loose money.

Somehow the hacker is doing me a favor by prooving what I was stating before they hired that joomla freak.
Also every minute I work on cleaning up the hackers mess is being paid.

The only weakness I know having on this system is the deactivation of openbasedir in php for some virtual hosts giving hackers the opportunity to walk arround on filesystem and take advantage of poorly chmoded files and directories (typical on a joomla install where they tell you to chmod stuff to 777).
While I was clearly against the requested openbasedir deactivation I cannot avoid it since it's on request from the client and they pay for the whole server. I think I did my part by refusing it for some time against their will but they said I was not to blame if anything happens.

Normally I just clean up the mess after such hacking incidents and don't care about tracing those guys since it's a waste of time and especially money.
But this time it's different, if the hacker is who I suspect him to be it's come down to a person who tried to make me look like a fool and take over my contract.
Prooving his identity would strengthen my contract with that company (that's why I am putting so much work in it).

Currently all I do for tracing is checking apache and ftp logs (yeah the guy managed to get FTP to 2 sites he worked on and they didn't change it after his freelance work) and hope he makes a mistake and forgets his proxying for once.

@prodev05 : it was a php script called C40 powershell, don't worry he doesn't have real shell access but only limited to PHP powers (but that can be enough for some things)

Can you explain how of all the people in the world who have the motive, opportunity and means of compromising your webserver that you've somehow magically narrowed it down to one guy?

The first time I found suspicious files on the server those file were just uploaded thru FTP and only me and him had the pass.
I am pretty sure I didn't loose the pass since the pass is sitting in a file with passwords to all other webspaces and none of those were ever used (also that file contains ftp accesses to more interesting machines and is sitting in a very very safe location).
So I cleaned up the mess and changed the pass.

After a few weeks my client asked for the pass because they needed someone to make some changes.
24 hours later weird files appeared again and when I asked them who received that pass and came down to him again.
So I changed the pass again and the same thing happened again. I am 100% sure the files came in thru FTP those times since logs don't lie.

Meanwhile, a couple of joomla site from that client where hacked and the files were identical to those uploaded via FTP before.

Because of that frequent "hacking" in such a short time I wrote a tool that scans all web folders every 30mins and puts all info about the files into a database (creation date, mod date, file size, md5, permissions, owner id, group id, ... etc) it also keeps track of previous versions of the files and even takes a "shadow copy" of the file (limited to 200kb per file).

This week a couple of weird files appeared again on some Joomla sites but no traces of hacking in apache access logs. Looked like the files just magically appeared.
So I queried the database with file information my tracking tool has builded and I found that there was a file on some webspace that apeared and was deleted 30 mins later (lucky for me just long enough for my script to track it). That file was uploaded thru FTP and when checking with my client only 2 external persons had that login information, me and again that same guy.
Again all the files uploaded were identical (even MD5 matched) and structure matched.

The hacker was always installing torrent servers (torrent flux) and his main downloads were naruto movies and I know for sure the guy I suspect is a very big naruto fan (he even got his youtube account banned for sharing illegal naruto content).

Also proxy used for all attacks were the same (one in africa with nearly no ripe info and another from SwissVPN)

Too much clues are pointing to the same guy.

You probably gonna ask how can he upload a file from one hosting to another.
Well that is simple, he uploads a hack tool with which he can upload files thru http.
Since I was requested to switch off openbasedir in PHP for the Joomla sites, PHP is not 'jailed' anymore and can browse arround on the file system (that's why I refused to do that in the first place) so if he knows the directory structure a little he can go to another webspace where some directory is chmoded to 777 and write files there.

11.0? That is no longer supported, so you won't be getting security updates.

BTW, how did you get your Joomla!? Directly from Joomla! themselves, 1.5.22 is current, and I can't see for 11.1 anything newer than 1.5.14 from repos (and 11.0 would, if anything, have been more antiquated). these could be old versions...

The feedback that I get is that Joomla! is OK out of the box, but that it is the modules that you add on that make it insecure (...primarily...mostly...up to a point, or something...), not having been as well written as Joomla! core. Of course, I am being told this by people who have a vested interest in that point of view...

This, of course, wouldn't be such a problem if you didn't need modules to make the website do something other than 'plain jane'.

Several comments, in no particular order:

• When I first saw the summary of this thread, and before I read in detail, my first thought was 'SSH'. So far, all that you seem to have to go on is a prejudice against Joomla! (which, admittedly, I partly share). Have you done anything to add to the conviction that the problem lies with Joomla! directly and not something else?
• blah, blah, blah...you this, the other guy that... Do you have any clear division of responsibilities with 'the other guy' as to who is responsible for what? For example, who checks through log files (and how often) to check for evidence of brute force attacks? Who has which passwords and are they really, really kept as secure as they should be? (and not, to be extreme, on a post-it on the other guys monitor). Who is is responsible for firewalling and ssh, and what approach did they take to resisting brute forces (or dictionary attacks)? The trouble with this two-guy set-up, unless you keep reviewing thing, there are a lot of cracks that something could fall down, and the 'low cost' stuff that you mention doesn't make it sound as if he schedules review meetings or keeps a watch on developing threats.
• And just in case it isn't 100%, already, and who is responsible for keeping Joomla! up to date and checking on any vulnerability reports?
• I will also suggest to you something suggested to me by the Managing Director of a company that I used to work for: "We do not let our Customers make mistakes". Of course, that is easier said than done, particularly when they see a mistake that they want to make, but if you protect your customers from their desire to do things that will cost them, in the longer term, then you have more grounds for them coming back to you. If you let them make big enough mistakes, they won't be around to come back to anyone.

This is getting a little off topic but just to respond to my Joomla prejudice.
Like you said the problem comes mostly from how it is used.
Lots of people just start chmod stuff to 777 to get it to work and when you asked them why they did it that way they just respond 'it was the first solution I found when I googled my problem' and when you ask them if they looked for more details like vulnerabilities that come from this solution you only get a moment of silence where you have to point out that it wasn't a smart thing to do.

Anyway, it's not my job to keep those Joomla sites up to date. Currently only those sites get compromised because they do not follow my requests to secure or update them.
All I can do now is keept the rest safe.

About passwords and SSH : I am the only one having SSH access and the password is a random generated password of 12 characters and every 2-3 month I change 4 characters, after about a year I completely change it. All passwords I give to my clients are random generated (10 chars for FTP, 8 chars for SQL but restricted to loacalhost access only). All passwords given to them are given to only one person thru emailand the mail is send locally (windows domain with exchange where I login thru TS to send them from my account to that user). If it comes to my attention that any password is ever exposed in some way I directly change it (for example if I see mail replies with history containing them, even it is only between me and the sender).