I need help, as soon as possible since my Linux box is quite unusable at the moment.
I was playing with it yesterday, and Opera crashed as it does occasionally, so i started konqueror to continue browsing, then closed it and started Galeon, which didnt want to start, instead my memory started jumping very high, and in 20 seconds, almost all of it (1GB, 500mb RAm and 500 Swap) was eaten according to gkrellm and top, so to prevent a lockup i quickly issued a reboot.
When the system came up after the reboot again, it wouldn't start syslogd, instead it froze and after a while started spitting a bunch of:
Out of memory! Killed process 59 (syslog)
So i thought i was hacked or something. I used a rescue disk and removed the execute bit from rc.syslog to attempt to start the system, but this time the same thing happened, except it was with lpd not syslogd, in other words a process always ate all the memory, and it was a different process. I also tried different kernels, but no change.
I then went ahead and wiped out all trhe parttitions except /home since i already wanted to reinstall for some time, so i reintsalled, booted, and configured most of the system, recompiled the kernel and booted the new kernel, went ahead and modyfied some rc. files and setup XF86Config, and noticed that when i type startx nothing happens:
root@localhost:/$startx

It just sits there. I even tried different XF86Configs, but nothing, althoug whne i press ctrl-c i can hear the disk working and after 2 secs or so im back at the prompt, so since i have a dualhead card and only one monitor at the moment, i halted the system and connected the monitor to the other port on the card, thinking maybe X is started via that port for some strange reason. I booted the fresh install again, and....
I got the old Out of memory: killed process 59 (syslogd) message, so im back to where i was 4 hours ago....
The only thing that i can thing of is that .xinitrc was tweaked (i didnt check it yet) and caused some buffer overflow or memory problem, i have no idea what to do to fix this, i can only use a rescue disk on that computer, and windows...
Please, does anybody have any clue what the hell is happening!? 1GB of memory eaten up in less than 30 seconds!
Thanks you in advance for any help you can provide
-NSKL

linux will randomly kill anything(lpd, syslog, etc) when it's desperate for memory.

you need to watch "top" and look at the MEM column to see which process is eating up the 1gig of memory.

Well, if i can't login, i can't monitor the memory usage, although one time when i did login (and only one time) it seemed hostname was eating all the memory, which is very weird. Anyway i reinstalled everyhing completely, butr im very curious to find out what caused this trouble...
thanks
-NSKL

Have you tried booting into single user mode?

Maybe a glitch in your memory? Or a bad stick? www.memtest86.com

That truly sucks though.

I thought it would be bad memory since i disconected both computers and let them collecting dust for two weeks while i was on vacation, but memtest86 didnt find any problems. I did boot into single user mode, looked through logs and found something like:
unathorized access from <IPADD> port(get_connect)
which is why i thought it might be a hack, but i still don't understand how it could mess up the system so bad, after all i was surfing the net as a user, not as root, which means limited permissions.
I reinstalled everyhting this time, wiping my /home too, and now everything is back to normal, bit im clueless as to what could have happened...
Thanks for the help everyone
-NSKL

Security forum is more appropriate for this kind of question.

http://freshmeat.net/projects/chkrootkit/?topic_id=43

Install chkrootkit on ANOTHER Linux and check the "buggy" Linux with

mount /dev/hda2 /mnt/tmp
./chkrootkit -r /mnt/tmp

Well, NSKL, i wan't a good present for a new year
But anyway, you know when I chkrootkit my system, it also looks up if syslogd isn't infected. In your case exaclty SYSLOG daemon was infected, it means it was a sort of a rootkit ( I think it was it or some type of worm ) used the syslogd to crash your system. Youknow it is quite logical to infect exaclty syslogd. Well you can read this:
http://www.cs.wright.edu/people/facu...on/obrien.html

Good luck