Hi all,
I am running the following:
Slackware 10.0.0
Kernel: 2.4.26
Apache:2.0.52
PHP: php-4.3.9

I am running also Sendmail, Myslq and sshd, which are new and pathced (1 week old)
Previously, the system was attacked, so i reformatted and reinstalled everything.
This week a strange thing happened twice. The system shutdownd itself. It is a co-location machine. I called the ISP guys and they told me the machine is off.
I examined all logs (syslog, messages,last) and found nothing suspicious, exept there are some strange records in the apache log at both shutdowns.
I am sending a sample - i mean the strange ^@ sign, where shutdown occurs. This is from the first time:

212.50.16.97 - - [24/Oct/2004:06:04:43 +0300] "GET /bex/rotate.php?siteid=44 h**p/1.1" 200 340 "h**p://forum.esh
212.50.16.97 - - [24/Oct/2004:06:04:43 +0300] "GET /bex/rotate.php?siteid=45 h**p/1.1" 200 342 "h**p://forum.esh
66.196.90.38 - - [24/Oct/2004:06:04:59 +0300] "GET /bex/presentation/BeX_files/slide0006.htm h**p/1.0" 200 9323
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ ^@^@^@^@^@^@
213.222.61.3 - - [24/Oct/2004:10:24:58 +0300] "GET /affiliate/images/see.gif h**p/1.0" 200 1233 "h**p://free.hit
213.222.61.3 - - [24/Oct/2004:10:24:58 +0300] "GET /affiliate/images/space2.gif h**p/1.0" 200 58 "h**p://free.hi


And here is from the second freeze:

213.16.35.130 - - [27/Oct/2004:13:17:10 +0300] "GET /eshop3/groups/clipart17la/4.0 (compatible; MSIE 5.01; Windows NT 5.0)
212.36.10.101@ * Jh**p://ppp.bubko.info/login.php?PHPSESSID=5cf2dce9c2270933c3524bf1e6fd2ceb  §
<
ôÛ Ÿ�? 3Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)
212.36.10.101e ' Jh**p://ppp.bubko.info/login.php?PHPSESSID=5cf2dce9c2270933c3524bf1e6fd2ceb  ›
<
õÛ Ÿ�? 3Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)
212.36.10.101  >h**p://banner.search.bg/bin/ifgen?_id=aa0&_s=0&_c=0&_n=noindex  š<
öÛ Ÿ�? 2Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
213.222.60.17“  
>h**p://banner.search.bg/bin/ifgen?_id=aa0&_s=0&_c=0&_n=noindex
}<
÷Û #Ÿ�? 3Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)
212.36.10.101@ *  h**p://ppp.bubko.info/cpanel.php
}<
øÛ #Ÿ�? 3Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)
212.36.10.101e '  h**p://ppp.bubko.info/cpanel.php ›
<
ùÛ $Ÿ�? 3Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)
212.36.10.101  >h**p://banner.search.bg/bin/ifgen?_id=aa0&_s=0&_c=0&_n=noindex  —
<
úÛ +Ÿ�? 6Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; DigExt) 80.97.148.8e C 9h**p://ppp.edemarts.com/list.php?dbpos=6&section=3&type=1
}<
ûÛ 3Ÿ�? 3Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)
212.36.10.101G *  h**p://ppp.bubko.info/search.php
}<
üÛ 3Ÿ�? 3Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)
212.36.10.101e '  h**p://ppp.bubko.info/search.php ›
<
ýÛ 3Ÿ�? 3Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)
212.36.10.101’  >h**p://banner.search.bg/bin/ifgen?_id=aa0&_s=0&_c=0&_n=noindex 

<
þÛ >Ÿ�? 3Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)
212.36.10.101A * ¾h**p://ppp.bubko.info/search_result.php?user=&fromage=&toage=&smoke=&town=&country=&aim=%CD%E5%E0%ED%E3%E0%E6%E8%F0%E0%F9% E0&search=%CC%FA%E6&sex=%C6%E5%ED%E0&pic=on&login=%D2%FA%F0%F1%E8%21 

<
ÿÛ >Ÿ�? 3Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)
212.36.10.101e ' ¾h**p://ppp.bubko.info/search_result.php?user=&fromage=&toage=&smoke=&town=&country=&aim=%CD%E5%E0%ED%E3%E0%E6%E8%F0%E0%F9% E0&search=%CC%FA%E6&sex=%C6%E5%ED%E0&pic=on&login=%D2%FA%F0%F1%E8%21  ›
<
Ü >Ÿ�? 3Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)
212.36.10.101“  >h**p://banner.search.bg/bin/ifgen?_id=aa0&_s=0&_c=0&_n=noindex  ‹
<

Ü AŸ�? 3Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)
212.36.10.101@ * .h**p://ppp.bubko.info/userView.php?userid=5108  ‹
<
Ü AŸ�? 3Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)
212.36.10.101e ' .h**p://ppp.bubko.info/userView.php?userid=5108  ›
<
Ü BŸ�? 3Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)
212.36.10.101’  >h**p://banner.search.bg/bin/ifgen?_id=aa0&_s=0&_c=0&_n=noindex  ƒ
<
Ü QŸ�? >Mozilla/4.0 (compatible; MSIE 5.01; Windows NT; YComp 5.0.2.6) 217.145.160.164e B 
h**p://free.bol.bg/popof/  s
<
Ü WŸ�? 3Mozilla/4.0 (compatible; MSIE 6.0b; Windows NT 5.0)
195.24.90.246A * 
h**p://ppp.bubko.info/  s
<
Ü WŸ�? 3Mozilla/4.0 (compatible; MSIE 6.0b; Windows NT 5.0)
195.24.90.246e ' 
h**p://ppp.bubko.info/  ›
<
Ü ZŸ�? 3Mozilla/4.0 (compatible; MSIE 6.0b; Windows NT 5.0)
195.24.90.246<  
>h**p://banner.search.bg/bin/ifgen?_id=aa0&_s=0&_c=0&_n=noindex  ›
<
Ü ZŸ�? 3Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)
212.36.10.101s  >h**p://banner.search.bg/bin/ifgen?_id=aa0&_s=0&_c=0&_n=noindex  ‹
<
Ü ]Ÿ�? 3Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)
212.36.10.101@ * .h**p://ppp.bubko.info/userView.php?userid=5015  ‹
<

Ü ]Ÿ�? 3Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)
212.36.10.1019 ' .h**p://ppp.bubko.info/userView.php?userid=5015  ›
<
Ü _Ÿ�? 3Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)
212.36.10.101‰  >h**p://banner.search.bg/bin/ifgen?_id=aa0&_s=0&_c=0&_n=noindex  ƒ
<


(There is more like this)

Any ideas appreciated

10x

i sont suppose you have snort installed ?
if so post those logs too.

10x for the suggestion,
I just installed Snort and am monitoring from now on
If it happens again - i will post logs
Greetings

I am running Snort now, but nothing unusual.
Saturday there were two shutdowns.The guys at the isp just say the machine was powered off - like after a normal
shutdown, but the database was broken and the startup log says the filesystem is reconstructed ( the filesystem
is reiserfs and i am using software raid mirroring)
There is nothing strange about snort. The most unusual record, around the shutdown time is:

[**] [119:7:1] (http_inspect) IIS UNICODE CODEPOINT ENCODING [**]
10/30-07:50:03.888883 66.249.66.205:65364 -> 195.34.96.130:80
TCP TTL:44 TOS:0x80 ID:31777 IpLen:20 DgmLen:347 DF
***AP*** Seq: 0x8403A5F3 Ack: 0xEBE10B3E Win: 0x84F0 TcpLen: 32
TCP Options (3) => NOP NOP TS: 341248673 23915244

The only anomaly i see is at the apache log:
66.249.64.27 - - [30/Oct/2004:07:52:02 +0300] "GET /eshop3/index.php?url=/eshop3/phps/menu.php&get_vars=1077* HTTP/1.0" 200 2626 "-" "Googlebot/2.1 (+h**p://ppp.google.com/bot.html)"
66.249.64.6 - - [30/Oct/2004:07:51:56 +0300] "GET /eshop3/phps/searchmachine.php?searchtext=CABLE-406 HTTP/1.0" 200 24091 "-" "Googlebot/2.1 (+h**p://ppp.google.com/bot.html)"
66.196.91.130 - - [30/Oct/2004:07:52:01 +0300] "GET /eshop3/phps/home.php HTTP/1.0" 200 12417 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp; h**p://help.yahoo.com/help/us/ysearch/slurp)"
66.249.64.52 - - [30/Oct/2004:07:52:05 +0300] "GET /sitemap/addord_1785054.html HTTP/1.0" 302 317 "-" "Googlebot/2.1 (+h**p://ppp.google.com/bot.html)"
194.12.255.166« ' [h**p://ppp.bubko.info/objaviView.php?msgpid=1410&PHPSESSID=51ed14ca9d78db1f6b46962548c6f1eb ¦<
sÚJ @¡�@ CMozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1) Opera 7.21 [bg]
82.137.126.99@ * 9h**p://ppp.zapoznanstva.dir.bg/objaviView.php?msgpid=1120  ¦<
tÚJ A¡�@ CMozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1) Opera 7.21 [bg]
82.137.126.99« ' 9h**p://ppp.zapoznanstva.dir.bg/objaviView.php?msgpid=1120  ·
<
uÚJ A¡�@ 2Mozilla/4.0 (compatible; MSIE 666.249.64.142 - - [30/Oct/2004:11:01:23 +0300] "GET /eshop3/sitemap/sitemap_mostviewed_0.html HTTP/1.0" 302 324 "-" "Googlebot/2.1 (+h**p://ppp.google.com/bot.html)"
62.204.150.100 - - [30/Oct/2004:11:01:24 +0300] "GET /affiliate/adv.php?key=cc782f66347392122e4ae25dca7e0dc4 HTTP/1.1" 200 1534 "h**p://kamasutrabg.hit.bg/pics.htm" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
62.204.150.100 - - [30/Oct/2004:11:01:26 +0300] "GET /affiliate/images/logo_text2.gif HTTP/1.1" 304 - "h**p://kamasutrabg.hit.bg/pics.htm" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
62.204.150.100 - - [30/Oct/2004:11:01:26 +0300] "GET /affiliate/images/space2.gif HTTP/1.1" 304 - "h**p://kamasutrabg.hit.bg/pics.htm" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
62.204.150.100 - - [30/Oct/2004:11:01:26 +0300] "GET /eshop3/images_products/pic70px/varta_ledlight2_1.jpg HTTP/1.1" 304 - "h**p://kamasutrabg.hit.bg/pics.htm" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"


There are some strange records around there.
I am starting to think there may be some hardware issues.
Please, give me some ideas

ehh..im not sure whether this is a great idea but anyway, i will give it this a try. If possible, try to disable the apache server and see whether this problem still arised. Generally, it is not a good idea to focus the problem on one service(apache) only. Look into syslog logs for more information. I had this experiences last few months ago. The server shutdown itself randomly. Few times per day and the syslog's logs showed nothing wrong and everything is fine. The server is patched and clean from intrusion. Further investigation revealed that it was the power socket problem

Hi,
Unfortunately i can not stop Apache and can not replicate the problem.
I am starting to think it might be hardware now - hdd or memory, maybe motherboard

I will post results if i find out something.
If anyoune has ideas, please post. I have not reached a solution yet.

Hi,
another development observed:
I started the command:
top -b > /var/log/top.log &
to check what is the CPU state in the time of crash.
Today, another crash occured and here are the last lines form top.log

top - 12:22:28 up 1 day, 14 min, 1 user, load average: 1.28, 1.32, 0.95
Tasks: 133 total, 2 running, 131 sleeping, 0 stopped, 0 zombie
Cpu(s): 8.7% user, 3.6% system, 0.0% nice, 87.7% idle
Mem: 904480k total, 878908k used, 25572k free, 133000k buffers
Swap: 2048248k total, 9124k used, 2039124k free, 526084k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
15997 nobody 12 0 7384 5172 3288 S 4.2 0.6 identd: No such file or directory
Oct 13 08:13:23 www inetd[422]: /usr/sbin/in.identd: exit status 0x1
Oct 13 08:13:23 www inetd[8128]: execv /usr/sbin/in.identd: No such file or directory
Oct 13 08:13:23 www inetd[422]: /usr/sbin/in.identd: exit status 0x1


The last 2 lines repat about 1500 times. Before this , i observe normal top entries.
I am not running identd. I stopped inetd after this crash to see what happens.
Can sendmail be using identd ?
Any ideas ?

Just curious...what happened to your server's date?

The server date was ok after the restart, just these messages show strange dates

check this link --> http://www.linuxquestions.org/questi...009#post910009

hope it got something to do with your problem.cheers!