Sitting at the console, I log in with any user name and NO PASSWORD IS REQUESTED. I get logged in automatically without entering the user's password.

I did:

passwd joeuser

to change his password and still he goes right in without being asked for a password!

What is going on?

Possibly related- 10 days ago, my smtp server was breached as a spam relay. The username they cracked was deleted. I added fail2ban for postfix. The logs show no further intrusion.

I would say you've been rooted. I would find a pre-intrusion backup and start again. Or preferably reinstall and only use the backups for replacing user files. not scripts or programs.

You don't say what version/distro of Linux, or what desktop environment and login manager. Also, if you SSH/telnet into the box over the network, does it prompt you for a password there??? And has it done this before from the console, or did you just notice it?

Some login managers can be set to allow users to log in with no passwords, at the console only. The theory being, your console is in a secured computer room, where only admins go, so it's 'safe'. I personally think it's a horrible idea. Smoker may be right on the money too, but without more details, it's hard to say which direction to go in. Since your box was breached previously, I'd DEFINITELY remove it from the network, and check for root kits and any other nasties, before continuing.

Thanks for your quick responses.

I am using Mandriva 2008 on this box.

I may not understand what you mean by "login manager". I take that to mean GUI but this is happening at the command line. I don't really use KDE except for the browser but kde is installed and running on this system.

It may be important to note that this box is one of 2 servers that we use, the other being in a remote location.

Regarding ssh, I usually keep login by password disabled. To test, I jut briefly enabled it and a password was required to get in.

No. This has never done this before to my knowledge. I don't run telnet or ftp.

I'm interested in this subject beyond my problem. How is smtp access leveraged into being rooted? Surely there are servers with thousands of smtp authorized users all of whom can't be trusted to wander around the server???

I read a little about rootkits which is a new subject for me. I installed and ran chkrootkit and had lots of nothing found or detected but at the end had the following:

Checking `chkutmp'... The tty of the following user process(es) were not found
in /var/run/utmp !
! RUID PID TTY CMD
! -DHAVE_VHOST_ALIAS! -DHAVE_VHOST_ALIAS! -DHAVE_VHOST_ALIAS! -DHAVE_VHOST_ALIAS! -DHAVE_VHOST_ALIAS! -DHAVE_VHOST_ALIAS! -DHAVE_VHOST_ALIAS! -DHAVE_VHOST_AL

From what I read, I really need to run chkroot from a cdrom which I will have to postpone until later.

Please never ever utter that phrase in this particular forum without proof.


You may have been doing this for over a decade but still it's the wrong advice if the OP has not completed assessment beforehand wrt the infection vector.


Better but still the same applies as above.


Best help the OP with the analysis, pinpoint quick wins like vulnerable application versions, lax access permissions, et cetera. And once completed only then move to the mop-up phase. A well-paced approach works best.

1 members found this post helpful.

Details please.


Nice but what was done to determine integrity of the OS and applications before "patching things up" as you did?


When did this start happening? Do you have system logs preceding that date?


http://www2.mandriva.com/support/lifecycle/ says your version no longer receives base system & components updates. Please do not use distribution releases past EOL date.


That could point to a PAM stack problem related to local logins. Just a hunch but could you diff your /etc tree (or compare MD5 hashes) with the remote machine? Focus on /etc/pam.d, /etc/security, the passwd and shadow files and also /sbin and /bin contents.


"-DHAVE_VHOST_ALIAS" is an Apache command line directive, see https://qa.mandriva.com/show_bug.cgi?id=26038 for an example of 'chkutmp' and Apache. See http://www.linuxquestions.org/questions/linux-security-4/chkrootkit-turned-up-problem-with-chkutmp-496382/#post2480887 (recycling a vintage 2006 post) about the "tty" part.

Thanks for the reply.

We run two servers in remote locations. While enroute to repair modem issues on server2 (3 day journey), users started complaining about no outbound mail service on server3. Checking server3 remotely, I discovered 100% disk usage on /var. The culprit was huge /var/log/mail/maillog file and /var/log/messages file. I don't know why this resulted in cutting off smtp users. The mail queue was also loaded with undelivered mail (3300 email) because the internet outbound server mailhop was over daily quota. The log files were unwieldy. They were taking over 20 minutes to load and the same to skip to the end. I chose to delete them to reinstate mail service quickly for the users and recapture disk space before further damage and unfortunately before I was able to see how it was breached. I cut off smtp auth for users outside the intranet and gave them direct access to the internet outbound mailhop.

After hours, I opened up the mail server for smtp auth and watched the log. I checked back through the night and when I saw the breach again, I cut off smtp auth. The log showed the server had been breached with the username and weak password of a particular user. Admittedly, I have operated the servers for years with weak passwords without trouble. I had used fail2ban effectively to stop brute force attacks and had no trouble. Unfortunately, when I opened up the outbound mail server to smtp auth about 6 months ago, I didn't realize that the fail2ban postfix entry needed tweaking to include a particular regex generated during a smtp auth failure that wasn't covered in my original fail2ban postfix configuration. Oddly, I had watched the mail logs before the breach and never saw the long lists of brute force attacks that one might expect if fail2ban wasn't working. I also received many daily fail2ban notices about banned ip's due to postfix attacks.

So, having traced the breach to a single user, I deleted that username and password. I did nothing to check the os or the integrity of apps. I didn't then and still don't understand how an smtp breach could be parleyed into access to the system at large. I also am ignorant about how to check for same.

The change in the login was seen first this morning, my first day after returning from the repair job at the remote server. Yes. I have logs going back but I have been checking them sereral times a day since I noticed the smtp breach. The first logs after the breach were sacrificed as described above. Maybe I shouldn't but I assume logs prior to the breach are not relevant?

Thank you for your comment on Mandriva 2008. I understand but I'm constrained by budget. I'm operating fairly old servers and I'm loathe to upgrade software because it often requires hardware upgrade and I have no budget. If you don't approve of Mandriva 2008 which I run on server3, then you certainly won't approve of Mandrake 9.2 which I run on server2. In fact, I ended up starting from scratch on server2 and tried installation of 2010, 2008, and 10.0 to no avail. I ran out of time and got scared I wouldn't have an operating server so back to 9.2.

Ok. Trying to follow your lead on the tree differences if I understand, I am looking for changes that might have been made on /etc files on server3 by comparing to a "model" which in this case would be server2. However, server2 and server3 have different distros. There are too many differences to enumerate. 410 directories 1869 files vs. 808 directories 7007 files. Did I understand the point of this advise correctly? Looking for files in server3 that may have been altered? MD5 hashes, if I understand your suggestion, would also be different between server2 and server3 and not necessarily indicate something abnormal because of differing distros. Am I at a dead end with this approach because of the differing distros?

I note /etc/pam.d/system-auth file has a very recent mod date 6/23. I don't recall making any changes there but I suppose a recently installed rpm might have changed it?

/etc/passwd and /etc/shadow...I don't know what I'm looking for. I see all the usual suspects in /etc/passwd. I see lists of encrypted passwords in /etc/shadow. Both files modified today because I changed one user's password as a test.

I also note that when I changed the user's password earlier today, it did not prompt me for the old password before the new. I tried a test; copied shadow to shadow.old then changed password for a user 2 times. No "old password prompt" at all either time. Copied shadow.old back to shadow.

Looking in /etc/security...don't know what I'm looking for. File mod dates all old.

Contents of /bin and /sbin no recent mod dates. What else?

Checking the referenced posts on the chkrootkit output, without completely understanding the posts, I surmise that the chkrootkit output I posted did not shed any light on the issue.

2006="vintage post"? We live in different worlds. I have OS's older than that.

Thanks!

passwd will not ask for the users passwd when run as root.

While unspawn would have you searching for an attack vector, this could take the rest of your life. Absence of evidence is not evidence of absence. In my decade of experience, mostly with internet facing hosts, one learns when to cut your losses and start again. If you have allowed weak passwords for some time, then you have probably not followed other security protocols with respect to preventing intrusion. For instance, you have only just installed chkrootkit.

A clean re-install followed by a thorough security lockdown before restoring public network access is the best approach, if only to get into a known state. It would have been best if it had been done originally, but them's the breaks. It's far simpler and easier to tighten up security on a known machine than it is to "mop up" after an intrusion.

If you can't fully convince yourself that's nothing's wrong on the current system, then you will always be looking over your shoulder.

Do you regard this as "the wrong advice" ?

It's your machine and your choice. Other people who post here don't have to deal with the problem, just pontificate on it at your expense.


BTW, for the nth time of stating, my sig is not a "claim to fame", it is a pre-emptive response to newbies who were basically (and insultingly) telling me I was wrong because they didn't like/agree with my answer (read - I didn't give them chapter and verse on a plate). People who continue to deliberately misunderstand it and use it to "win" arguments need to get a life.

0 members found this post helpful.

FWIW events listed in chronological order as far as I can place them:
- opened up the outbound mail server to smtp auth about 6 months ago
- operated the servers for years with weak passwords

- 10 days ago, my smtp server was breached as a spam relay.
- (..) no outbound mail service on server3.
- 100% disk usage on /var. (..) /var/log/mail/maillog file and /var/log/messages
- mail queue (..) undelivered mail (3300 email)
- delete them (:logs)
- cut off smtp auth for users outside the intranet and gave them direct access to the internet outbound mailhop.
- when I saw the breach again, I cut off smtp auth.
- The log showed the server had been breached with the username and weak password of a particular user.
- having traced the breach to a single user, I deleted that username and password.
- used fail2ban (..) fail2ban postfix entry needed tweaking to include a particular regex generated during a smtp auth failure
- received many daily fail2ban notices about banned ip's due to postfix attacks.
* did nothing to check the os or the integrity of apps.

- change in the login was seen first this morning (:19/Jul/2010?)
- /etc/pam.d/system-auth file has a very recent mod date 6/23
- when I changed the user's password earlier today, it did not prompt me for the old password before the new.



Only "quick wins" like major differences in user accounts, user authentication.


Until this post it was not clear that you run different and deprecated distribution releases. If the RPMDB can still be trusted (compare with a copy from a backup) then running verification with 'rpm -Vva' on the "victim" might help. (In a worst case scenario, if the RPMDB should not be trusted and if you do not run any file system integrity checker, then you could still compare with packages from a trusted repo.)


Only "quick wins" like any added users, any users that should have their account locked, any users with rights to wheel group, any users with root rights.


That can be a PAM thing but it can also be Something Completely Different.


Good. See below.


No. Best *attach* full logs.


The key here is to not make assumptions but make certain. Since PAM stack files are plain text it would be easy to 'stat' it, then review it (or compare contents with the file from the other machine or compare with package contents from a trusted repo). To get a list of packages by install date use 'rpm -qa --qf='%{INSTALLTIME} %{NAME}-%{VERSION}-%{RELEASE}\n';'. The first field (date) is in Unix epoch making it easier to sort.

If this is not a package update then, if you are the only admin for the machine and if the file permissions were not skewed, you have bigger problems on your hands as changing it would require root rights. If that's the case you should halt all maintenance and testing, read http://web.archive.org/web/200801092...checklist.html, then close off public access and then perform those checks. Also if you made backups and if those backups contained copies of system and daemon logs it could be beneficial to extract the logs to a local workstation and run 'Logwatch' on them as it makes pinpointing problems and generating leads for investigation easier.

Hi Smoker. Just Hi.

I'm a self taught guy starting with punch cards in 1975. I'm not bragging, more like apologizing. Like a lot of self taught people, I have gaps in my knowledge and procedures and I tend to focus on learning just what I need to learn to get to the next level. It has worked well for me. I'm a business guy but I love computers and always have just like most of the people who thankfully volunteer their time and effort here. Linux itself is the same kind of volunteer cooperative effort. It blows my mind how voluntary cooperation (without slavery) has produced the most amazing product since the great pyramids. It also amazes me that so many people are willing to share their knowledge and expertise for free. I've read many many posts and often see people asking questions as if this were some kind of pay per answer service; demanding this and that, just like you said, on a plate. Questions should be asked and answered in the spirit of cooperation and curiosity that spawned Linux in the first place. The people ask the questions, the answers give direction and clues, the users man and google until they're stuck again and then more clues. More often than not, the answers are triggers rather than actual "answers". I can see how it can get old for those with the expertise, sharing and giving their time and advice without compensation or even gratitude in some cases.


I thank you very very much for your sincere and well intentioned advise. Your approach makes complete logical sense to me. The feeling of looking over your shoulder is not unfamiliar to me over these past 3 weeks. And I have come to understand "cutting losses" many times before. A reinstall is not the end of the world. In fact, when I think of how much I've learned since I first installed, I welcome the opportunity to get it right (or at least better) the second time around.

I'm not real scared or nervous about my current system status. I've been through hell with users the past 3 weeks. They are now happily catching up on their work and again singing the praises of the system that replace volumes of books, ledgers, records, files, logs, lists, typewriters, carbon paper, copy machines, pencils, calculators, calendars, armbands, rubber stamps, green visors and large erasers. Every night I make a careful and safe backup of their work because I owe that to them particularly right now.

But again, the subject interests me, in and of itself as pure knowledge. I don't understand how it was done. I have not seen it before and have yet to hear anyone say they have either. Yes we've seen hacks and breaches but I've read many posts here and elsewhere from people who are trying unsuccessfully and intentionally to do the very thing that was done to me. Many responses come back to them that it can't be done but it obviously can be done. That makes this an interesting question.

So your points are well taken, registered and accepted. You've fulfilled your 'obligation' to the question. Yes, your signature may be slightly off-putting to some but it doesn't bother me in the least. Your confidence makes me chuckle a little and I think, "good for him".

So for the next few days, I'll satisfy my curiosity and go down the roads Unspawn has laid out to take a deeper look at what might have happened. Hey, Unspawn is heavy duty. Lots of tenacity, lots of technical knowledge. I want to learn more and its a good opportunity. Have you ever had your house burglarized? You want to know, how did they get in? Sure, the police are there, you'll get insurance but you still want to know, how did they get in?

Come Saturday or Sunday, when the users are gone, I'll make a decision, for right or wrong whether I'm reasonably safe or whether I need to start fresh.

You're a good person smoker, very cool, thanks for the advice.

Unspawn-

Thanks for these suggestions and clues. It will take me a while to come to some kind of understanding and see where they lead.

I'm very curious about what if anything was done to me and how.

If it wasn't for security requirements, computers would be a fairly simple matter, don't you think?

Anyway, good stuff to do and try. Thanks.

True. It's also proof I'm not infallible.


Such a response only shows disrespect and a lack of understanding for what we try to do here. As your replies here warrant a more verbose response without making this thread go OT you'll find it here.

Feel free to post your progress or ask for more details wrt tools and actual commands. And if you would like help with assessing bulk data feel free to drop me an email.

I've made some progress as follows:

A mail loop was accidentally created this past week when a user was going on a trip. In order to stop it quickly while I figured out where it was coming from, I changed the user's password on server3 temporarily. Guess what? Access was not denied to pop3 even though the user was now using the wrong password. So terminal and pop3 were open which lead me to focus on a common denominator, PAM, as suggested above.

Reading up and following up on PAM, comparing /etc/pam.d files on server3 with server2, I discovered some changes in /etc/pam.d that I didn't make. It may not be appropriate to post the details. I eliminated the changes I found and the pop3 hole appears to be fixed. I am not sure yet whether the terminal hole is fixed.

Files in /etc/pam.d were all 777. I changed them to 644.

(background to above: I made a disastrous mistake a long while ago by accidentally changing all /etc files to 777. I had changed them back to the best of my ability back then but I could easily have missed this directory. No harm ever came of it until I opened up smtp service.)

I also note that the permissions on /etc/shadow were wrong. I changed to 600. I cranked up an old server called server1 to have another comparison. I note that permissions on /etc/shadow on server1 were 600, on server2 were 400. Owner and group on both are root. Does it matter 600 or 400? 400 seems strange to me. How does root change someone's password if it doesn't have write permissions to /etc/shadow?

I'm sure there's more mopping up and checking to do. I'm going to follow up on other unspawn suggestions above.

Something fundamental still bothering me....

The original hack came through smtpd. So, the hacker guessed a user name and password because fail2ban was not fully functional on smtpd at that time. Even armed with a username and password, how did they get in to make changes in /etc/pam.d? I've run sshd without password authentication for many months. I don't run telnet or proftpd. I do run webmin but there would have been no access for the compromised username and password. Logs do not go back to 6/23 which is the mod date of the changes in /etc/pam.d.

Sorry for the late reply. If you could please post your changes as PAM configuration files do not contain privacy-related information unless non-root specific account names are used in configuration lines which would be rather uncommon.


Checking the terminal hole would just be checking differences between PAM PAM configuration files and testing it.


And thus the value of regularly running a file integrity checker (Samhain, Aide or even tripwire) on the system. Also RPM-based systems have 'rpm' commands allowing you to 0) verify permissions (-V) and 1) restore default permissions (--setperms).


I just hope that statement was backed up by lack of evidence in logs and the file systems.


Where other root-owned processes need to read /etc/shadow the permissions may be set to octal 0440 but AFAIK by default /etc/shadow should have octal 0400 permissions. This does not impede root in any way modifying it.


The basic admin stance towards users should be that no users are good users. (It's for exactly that reason that some services, like for instance Vsftpd, allow for virtual user accounts bypassing all the trouble, restrictions and auditing that go with enabling "real" accounts on the system.) Most warding off of Evil happens during the connection setup (firewall, service configuration, (X)inetd, tcp_wrappers) and authentication phase (service login, system login, PAM), so once an authorized user gains foothold most systems are left wide open without much logging (other than login records, process accounting and shell history files) and without limiting time spent on the system, logging commands that should not be regularly executed, setXid commands freely accessible, et cetera. Practically speaking this means any user is free to reconnoiter the system. Add to this the fact you "operated the servers for years with weak passwords" and you change of DAC permissions this then offers non-root users unlimited access to about any part of the system.

To cut things short I think your initial question of "how" will be difficult to answer, partially because GNU/Linux out-of-the-box does not enable auditing facilities like one would want to see to support assessing situations like these and partially because you deleted system logs. Armed with more information I could assist in a more thorough investigation of the machine might (it might reveal some clues, but no results can be guaranteed as the system as seen way too much use since the initial breach so I wouldn't ever label it forensically sound) but the trade off is it will take much of your time and effort. Since your OS release was unsupported, outdated and EOL'ed ages ago I would suggest setting up a staging area machine (or use virtualization) on a different network, install a supported release of an Enterprise-type distribution, configure and harden it, then scrutinize data and accounts before migrating them. While initial setup and migration will cost you time and effort in the end a "clean machine" approach (from a systems integrity as well as a business point of view) will be cost-effective, ensuring you can provide services securely and for many years.

1 members found this post helpful.