I have a hosting account on a shared server and my index.php files keep getting infected with a line of a iframe on the first line afte the <?php. This, is loading a malicious web page through my websites.

Now this is not an isolated event - it's keep doing this every 15 min as I managed to discover so I made a bash to run through a cron job every minute now to remove that line from my index files(I know it's server killing but it's a desperate solution)

Now my general question is: How the heck is he doing that?
More specific: How can I find out what process, script or user is editing my files.


More on that:
- Once the "thing" is editing the files the edit time from file properties remain unchanged and only changes when I do the editing (so the only thing that changes is file size)

- my Linux BOX 2.6.32-46.1.BHsmp #1 SMP Tue Sep 6 12:18:02 MDT 2011 x86_64 x86_64 x86_64 GNU/Linux
- I tried disabling SSH and found out that the "thing" happened even with my SSH turned off
- Tried changing most of my passwords
- The hosting company doesn't give a sh1t
- I'm a linux newbie but I'm not a stranger from programming so I can do a bash or two and I need desperate help.

change password of ssh, bind ssh to ip /certificate,close ftp!
check running process list ps -A or htop for suspicious processes.
then remove all files and left only index.php in root of site
if after all this file is changed a) rootkit installed b) server hacked
a) rootkithunter/reinstall sys b) complain to service provider.
check .bash_history in home folder,apache logs, ftp logs, install snort/fail2ban

0 members found this post helpful.

btw you can save logs in rt on remote linux box

@OP: I rarely see need to intervene but until a proper reply is posted please disregard post #2 as it disregards proper discovery, shows no logical order to tasks to perform and mixes all heaps all sorts of things into a confused multitude of probably well-meant "advice".

1 members found this post helpful.

Lets start with something simple. What, or who, is the owner of the index.php file and the directory that contains is and what are the permissions of this file and directory?

The second thing that I would recommend is an experiment to determine if this action is being caused by something local on your machine or is being caused by external activity. Since you are running in a hosted environment, I would ssh into the machine and then activate the firewall to block all other traffic. Then give it a sufficient period of time, in your case 15 minutes, to see if the changes occur again. Normally when performing an investigation, one of the first steps is to isolate the machine to avoid tipping off an intruder and to help minimize alterations to the crime scene. This experiment would be this first step, but also performed over the short run, would also appear as if the system went down for a few minutes, which can happen for several reasons. Whether or not to put the system back on line would be up to you, but given what is happening you should consider proceeding with the investigative process, in which case you should leave the machine isolated.

The CERT intruder detection check list has a set of steps used to investigate potential compromises. Amongst the items in this list are actions of looking for hidden files and modified files, and examining what processes are running. While you have the machine isolated, I would run these commands.

It is interesting that you are not seeing the time attribute modified, except for when YOU are making the changes. This could be an important clue, but I am not sure what to make of this discrepancy at the moment. My initial reaction is that the attributes of the file may have been changed, but I am not sure how as to account for this behavior. However, when you do check the file, in addition to the permissions and owner, I would also like to know what the file attributes are. I would suggest using the ls -i command to view the inode attributes, which will also give information regarding links.

"What, or who, is the owner of the index.php file and the directory that contains is and what are the permissions of this file and directory?"

Like I said before I'm on a shared hosting server therefore I don't have root powers - on my account I host several websites - like 20 - all with their specific name as directory in ./public_html like ./public_html/site1.com/, ./public_html/site2.com/ and so on. Now all my ./public_html/site1.com/index.php - like files have been affected but wihtout being recursive so ./public_html/site1.com/scripts/themes/index.php was not altered. - Basically he altered only the websites homepages.

[~/public_html]#ls -al

drwxr-xr-x 8 myuser myuser 4096 Nov 16 13:04 site1.com/
drwxr-xr-x 12 myuser myuser 4096 Nov 16 13:04 site2.com/
drwxr-xr-x 6 myuser myuser 4096 Nov 16 13:04 site3.com/


[~/public_html/site1.com]# ls -al
-rw-r--r-- 1 myuser myuser 4998 Nov 16 13:01 index.php


[~/public_html/site2.com]#ls -al
-rw-r--r-- 1 myuser myuser 4976 Nov 16 13:01 index.php

[~/public_html/site3.com]#ls -al
-rw-r--r-- 1 myuser myuser 73886 Nov 16 13:01 index.php

"ssh into the machine and then activate the firewall to block all other traffic" - Can I do that without root access ? (I will google it myslf if not an easy task) If yes, how ? And will that mean that my websites will go down?

"avoid tipping off an intruder" It's a battle ground down here not a crime scene - For the last 3 days I am scanning the host from my ssh - I am building bash files to cron job them to detect when the file has beed compromised by running a find every 5 minute - dumping all that info into a file, I have files called infected.txt from my previous scans, ps -ax dumped to files so I can search for clues and so on.

Followed your advice I begin reading http://www.scribd.com/doc/6398/Cert-...tion-Checklist and the first line "Examine log files for connections from unusual locations or otherunusual activity." made me smile and remembered sunnydrake saying to "check running process list ps -A or htop for suspicious processes" Why? Well, if you've told me to look for suspicious processes on my win box I would tell you in a spot who doesn't belong there as I do it on a daily basis but to look into ps -A processes list and spot the awerdness for a linux newb is almost impossible - not to mention the fact there are showing all users processes or and they are like 100.

Some basic IDC I've done it myself by reading through the .bash_history file and nothing wrong in there - only me in there. That was the reason I told my customer support it might have been something with higher access than mine but they shutted my mouth by saying it could also be a (php) script running somewhere on my websites and it doesn't even need to log through ssh or have root privileges - and they are right.

This is why I will continue reading the IDC even if it's pretty technical with no lovely footnotes and I stick to my question - Is there a way to spot who (what process or at least user or anything) is editing my files ? - like monitor them somehow knowing all those facts.

I've started monitoring my index file properties by doing ls -as >> site1com.txt so I can spot any difference and as "we speak" it happened again and just spotted the fact the the time changes 8910 Nov 16 13:10 index.php to 8895 Nov 16 13:52 index.php. Also monitoring index numbers with ls -i .

PS: While I wrote that I stopped the cleaning bash cron job for monitoring purposes.

Without root privilege, you ability to diagnose the problem is going to be limited, as I am sure you are aware. This is both a blessing and a curse. It is a curse because you will be limited and it is blessing because there is a fairly finite number of things you can do in response before your provider must step in. Most of the steps in the IDC, you won't have permission to run.

First off, it is good that you caught it while it was happening as this showed that the time attributes are changing, which puts it more in the realm of an overwrite or file file modification. The idea of it being the 2nd line after the <?php line suggests make me suspect that you are dealing with a script that looks for that tag and then inserts some text, which would be near trivial to do with a PERL script or something similar.

Now with the clarification that your site directories are owned by you, as your own group (with only group read), this means that either the script is running under your user name or is running under root, both of which will have write capability to your files. Other users, including other group users, and locked accounts will not have this capability. Consequently, I would use the find command under your directory tree to see if there are any hidden files.

Lets start with the following commands:
This should show if you have any hidden files or root permissive files in your path. If you do, examine them very closely and carefully. You will have some, mostly under your home folder, like .vimrc, .Xauthority, and so forth. Unfortunately, the culprit is unlikely to be named something something obvious like "badScript".

You have also mentioned creating a cron job to restore your files. I assume that you didn't notice anything odd in your crontab? If this is happening every 15 minutes with high regularity, it is possible that it is being called by a cron task, but not necessarily yours.

2unSpawn well i see term "proper discovery" but no meaning of "proper discovery" in a post? My advice was based on discovery process, shame you not see it. Btw what i really missed to describe is web side crack, but on shared hosts very popular is to crack into other accounts from neighbors/root access.. next is direct crack of account via ssh/ftp. My advice was used to quckly remove possibility of web side hack and moveon second stage ssh/ftp/process. but yep little straightforward

2Noway2 firewall block is dangerous one wrong move and you can lock your own access to box. CERT doc make point on root crack but there is other possibilities.

to catch intruder partial actions only reliable way is to save logs on remote pc in log append scheme(or honeypot), next is masked process that logs data. I read some rootkits/cracktools code and usually they include clean logs/actions code. But i think what you really need is to get box working again and kick intuder.. if you want trace him it will usually lead to some botnet(90%) or another hacked terminal.

ps -A is needed.. 1st you need to know what programs is currently executed.. (and pay attention to root,ftp,apache,php and your login name). Remember what i said about neighbor crack? this information is needed. BTw if you timestamp ps -A logs you can see if process is spawned in time of modification.



lsof - list open files

I agree, it can be dangerous as you can lock yourself out. Without physical access to the machine, though, it is the only way you can safely isolate the system while removing the ability of the intruder to notice that they have been made and cover their tracks.

I suspect that there is little, forensically speaking, that a non privileged user will be able to perform. Unfortunately, their service provider has been less than helpful and said, "its not my problem." which couldn't be farther from the truth. At this point, the best approach I can think of is to either find the source of the changes or show that the compromise involves activity above the user's level, in which case he has proof that it is the responsibility of the provider.

You are correct here. PS and LSOF may help. Indeed, depending on what the file find shows, they would be the next steps to take.

find ~/ -user root -perm -4000 -print
and
find ~/ -name ".. " -print -xdev

Returned nothing but:
“warning: you have specified the -xdev option after a non-option argument -name, but options are not positional (-xdev affects tests specified before it as well as those specified after it). Please specify options before other arguments.”
And the third:

find ~/ -name ".*" -print -xdev | cat -v

Returned a lot of files most of them .htaccess and things like .1.php.swp, .bashrc, .gnome2, images/._arrows-666666.png, images/._favicon.ico, .info@mydomain.com, /.fantasticodata.OLD.1272976014.30230, so on .. picked up the most ^^ suspicious.
Could list the whole output with certain masking if relevant.
As for the cron jobs check – I am not allowed to use this program (crontab) and in my panel seems there is no other cron job open besides mine.

@Noway2 “At this point, the best approach I can think of is to either find the source of the changes or show that the compromise involves activity above the user's level, in which case he has proof that it is the responsibility of the provider.” This is exactly what I am looking for – the source of those changes or at least proof that actually involves activity above the my user level aka “it’s not my business”

As for the ps –A if I understood correctly you’d like me to “somehow” run this while the “thing” is editing the files and then compare with an earlier state of ps –A to spot the process that’s doing all that but it’s nearly impossible as I see the 15 minutes loop was not as accurate as I thought and that process could take less than 5 sec to run.

After monitoring with ls –i command I noticed that index numbers do change after his and my editing – what this could mean ? Do those numbers mean something or I can link them to something or are those just indexing numbers and that’s all ?
In the end I’ll try to monitor with ps –A or better make a cron bash script to do just that every minute and dump the result int a file but like I said .. there are 55 seconds when he could run that without being noticed.

My opinion is that even if he doesn’t (anymore) has access to my ftp or ssh he has somewhere into my 20 websites a perl or php script that he is running from a browser so is using my user privileges and is making those changes.
Now tell me if I’ve understood all this right. If there is indeed a perl/php script doing that I should see it with the ps –A command if I manage to somehow run it exactly when the per/php is running?

still imho you need to remove possibility of webside script crack first(before you monitor process list, e.g. you clean or disable all other files in web page / except index.php).. here is a command that will dump process list in files that you can latter check via diff.

watch -n <seconds> <command>

watch -n10 'ps -A > "file_$(date -R)"'

but tuneup of ps output recommended to be useabe with diffs(no cpu load,runtime,etc)

I think you need two simple steps to first stop the script from executing and 2nd find it.
1. Permissions permissions permissions.
Since you're in shared host and own everything inside your account: starve the script by denying write permissions on "index.php" such that even the owner will need to grant himself write permissions prior to execute this bad script(additional step).
chmod ugo-w index.php.
The web server just needs "read" permissions most of the times.

2. Find the script/file that has "index.php" inside it -since that is the file its writing to it will have to have that name inside the script itself. So you search in a given folder say(Document-root ) of your website and spit out all files that have "index.php" inside it. I pasted the code for you
Note: you will either have to run this script from the target folder or replace the "dot" after find and type the folder name(e.g public_htm).
Also, note for this script, i'm only searching for files of extension .sh. You can replace that with .pl for perl and .php


#!/bin/bash


for i in $(find . -type f -name '*.sh'); do


egrep "index.php" $i

done

what not simply ask hosting sysadmin to chattr -i the file, then see what happens??

OP, do you have type & versioning of the webserver stuff (joomla, apache, php, ecomm package, etc)? there are a slew of iframe injection vulnerabilities out there (OS Commerce ecomm stuff being a recent flub).

what tools are available to you (top, rpm, others)?? what filesystem(s) does it have? not knowing what you have makes this a wild goose chase with dead-ends everywhere.

imho, if this is a hosted package then the hosting provider should be dealing with the security issues!

Can I make a plea for enhanced readability here (use of quote tags, code tags, particularly) , please? When I initially read the post, reformatted below, my first read through gave me the exact opposite interpretation than the one intended:

And one (small) piece of actual help;

• problems like this can often be cleared up by reading the man page
• ls -i lists the inode for each of these files; essentially, this is an arbitrary number that the OS uses to access the file, but if someone takes the original file and overwrites/re-writes it, it will get a new arbitrary number

So, whatever the evildoer/evil process exactly is, if you see that number change, it is a sign that the underlying file is being changed.

You are saying that you believe that the evildoer is, in effect, using your login and not, eg, the log in of root. Have I missed the evidence of that, or is it just what seems likely to you at this point?

And, by the way, are those file names evidence that you are using fantastico (which I've never used, but has a bad rep, in some quarters *)? So, I guess, if you are using fantastico, or have it available, I'd like to be sure that it is an up-to-date version, given that it is pretty sure that older versions will have problems of some kind, although that's by no means the first thing that you should be doing.

* Roughly 50% of the people that I've spoken to about fantastico claim to have had big problems with it, the others claim that it is very useful, if you use it carefully, only do the right kind of stuff with it, and are careful (without ever defining very clearly what will work, and what won't). For that reason, I've always thought it easier to give it a wide berth, but that's just me.

1 members found this post helpful.

The .bashrc file is probably normal, at least if they have not been changed. The others are up for grabs. You will need to examine them carefully, noting that each of these was "hidden" from a normal directory listing. One thing to remember is that unlike Windows, Linux does not use name extensions to indicate file type. Instead it relies on the data in the first part of the file to determine the file type. Consequently, it is entirely possible to label a file xxx.png, make it executable and have it be a bash script. In any case, it looks like the list of files has identified a potential list to look at. You could refine the list down by using the -mtime or -ctime flags to eliminate files that existed before your compromise.

As far as the other techniques, you could watch the process list output. This will undoubtedly be an eyeball intensive process. As has been mentioned, in order to "write" to thee files, one of two things needs to happen: *** either the script is being executed by a user ID with sufficient WRITE permissions on your file, or the script is running with root privilege. Since the files are owned and writable by your user, changing the permissions to eliminate write access would give you a possible clue as to whether or not this is a root level compromise. What I mean by this is, if you files are not write permissive and they still get modified, chances are you are facing a root level problem, which would definitely go beyond your ability to address. The trouble with this approach is that you may be able to make them non writable, but I am not sure if you can restore this privilege without root access. I would try an experiment first, but a safe approach may be to make a backup copy with write permissions enabled and then restore the file by deleting the non-writable one and replacing it with the writable original.