This is what I have figured out so far:

My office runs RedHat 9 and we noticed that our internet services were slowing to a complete crawl and there were huge amounts of polling messages on the local network.

This box is mainly used as a private web host for our sales staff to get product information, for the most part, this box sits doing nothing most of the time.

After hunting through network messages, I tracked it to a little program in both /bin and /sbin called '(swapd)'. As a test, I renamed the file to 'badswap' and it seemed to solve the problem.

I did notice that during startup, as soon as the system said "Entering Non-Interactive Startup" the following message would start showing up repeatedly 'sh: 1 (swapd) command not found'. As far as my office was concerned, that was fine.

A year later, almost to the day the problem started again and sure enough it was the same file. I renamed it but this time it came back the following morning.

When looking thought the GUI Services list, I noticed that the status' showed the same sh error message on many but not all of the running services. The error message also shows up on shutting down so it seems to be called by something that is called during the S and K rc scripts but I have had no luck tracking it.

Any ideas?

Information about swapd:
http://freshmeat.net/projects/swapd/

Please, don't run such an ancient and unsupported OS. Security updates aren't available. It would be a lot easier upgrading to Fedora or RHEL than re-building all of your vulnerable programs, services & libraries from scratch.

I am quite aware of this project as I investigated it to start with... This is not what has been inserted on the system.

If it were up to me, I would likely have done this years ago, but it is not my decision to make. My job has become to keep the current system running as is, so how about something helpful and not quite so dismissive as that would actually be helpful.

Sure, you've already posted about "polling messages" (but what messages exactly?), the box being "mainly used as a private web host" (mainly, so there's more: what services are exposed over the public interface?) and this happening a year ago (why on earth wasn't this investigated properly back then?) but nothing tangible. So do yourself a favour and run 'strings -an4 [subject] | egrep -i "(pack|sock|promisc)"' on your saved copy of the foreign entity. If it returns anything, then 'swapd' really is a sniffer and you may consequently consider the machine compromised, at which point the machine should be brought down for investigation. Since somebody basically ignored the same warning a year ago there are no rational arguments why the investigation should not be dealt with 0) swiftly and decisively and 1) should not include all other machines on the network.

In short:
- Read before you act (if you don't know where to start see the Intruder Detection Checklist (CERT): http://www.cert.org/tech_tips/intrud...checklist.html or http://web.archive.org/web/200801092...checklist.html if (made) unavailable),
- Notify users their passwords may have been compromised,
- Log and deny all network access to the machine and do not continue using it,
- Make a bit-by-bit copy for investigative purposes if you didn't already,
- Halt the machine while you investigate and repeat procedure on adjacent machines until they have been verified to be clean and trusted.
* Just to make this efficient and clear: if any machine is found to be compromised there will be no question of "patching things up", "restoring" or "fixing things".


Doubts, questions, just ask.

Sigh. So what you're saying is that an unknown binary performing suspicious actions has been sitting on your internals sales systems (and, presumably, that network is trusted by your sales staff who are taking and storing monetary orders on their computers), for over a year, and *nobody* sees the problem with this?

"If it were up to me..." "My job has become to"... YOU are looking after the system. If that system is truly infected, or even suspected of being so, it is YOUR JOB to sort that out. Guess who guess the sack if the details of every sale your company has processed for the past year turns out to have been sold to your competitors or random people on the Internet? YOU.

Yes. Turn the system off. Explain that you suspect it has been compromised. Refuse to turn it back on until a replacment has been built and the data sanitised and transferred. If they FORCE you to turn it back on, make them sign a letter to that effect (they will HATE to do this and this is how you get what you want - no-one will want to sign an authorisation like that, so you will have pressed the fact that the system NEEDED to be taken down).

I have *done* this, on production systems, in the middle of the day for similar reasons (although, in my case, there was a suspicion that a system could have JUST been infected a few minutes before).

FFS... how hard is it... just from the information in your posts, I can tell that you believe this system is running rogue code on a trusted network - all that has to happen is for the author of such rogue code (who probably has complete control of the machine) to place exploit code inside the sales network. So you somehow think the best course of action is to leave it running for a year or so, then maybe casually try and kill a process and maybe raise an eyebrow when it comes back, then try to post to a forum to see what other people think etc.

PLEASE... tell me the name of your business... I need to check that I've never had any financial dealings with you.

Thanks unSpawn and ledow for your help and I am already doing most of this...

I have reason to believe that it was something inserted by a disgruntled employee to just cause havoc on the network, nothing truly malicious. It, (dswap), spends the majority of the time asking for machine ID's as if it were trying to DoS on the local network and requests for DNS from the internet -- nothing else from the packets I have read, and they have been many...

The part that I found interesting about the way this attacks and nobody seems to be interested in is that it seems to have placed itself somewhere in the startup system that gets called on every service starting AFTER the system finishes the interactive startup and immediately once the system enters "Non-Interactive Startup."

As I said in the original post, it runs in what appears to be the rc K and S files, like a library include that is called during their initialization. I would love some help finding where the systems scripts change at that point as I have had no luck looking through the rc scripts manually so far.

I just happen to be the one person who knows something about *nix in the office, the maintenance of this machine is not my usual job. I feel SO 80's bad cheesy movie.

Look, this may sound harsh but you haven't posted anything tanglible. Simply said: for those that actually do have experience with (or have a more than average interest in) incident handling or forensics your "story" holds nothing. If you just want to vent off or show off your prowess, I'd point you to using a web log. If you want help (and I must point out we *really* are a helpful community here), post details or questions we could actually help you with.

It's not interesting at all... a program is running on your system. Off the top of my head, I can name at least eight or nine ways to get a program to execute on boot automatically in Windows without even trying (I even know the relevant registry entries / folder locations off-by-heart) and even more under Linux.

It will be being called from *somewhere* - a cron job, at job, another process that's been compromised, maybe even an automatic remote compromise of one of its services (e.g. a virus on the local network that constantly re-infects that machine), possibly even just a simple wrapper script somewhere. It will not just be magically appearing. That's why *nobody* is interested in where it is... there are a million places it could be hiding and guessing them all one at a time is a pointless exercise because you *will* miss several and it could take weeks to find anything like that, even working at a decent rate of knots.

The time to tell HOW it is doing that is when the system is OFFLINE. In fact, in order to do it properly, you need to put that hard drive (or a byte-for-byte image of it) into another machine that *doesn't* boot from it and analyse it. It's easy to "fake" scripts that hide the presence of a process, or its executable, it's easy to "hide" programs inside other programs (just about every virus does this), compromise boot sectors, it's easy to "hide" files from the filesystem view of even the root user if you have enough access. While you are running a potentially-compromised machine, anything you do to try to find the compromise could mean NOTHING... it's like Anti-Virus auto-disk-scanning on Windows... by the time your antivirus could see a virus, it's already too late. 90% of the viruses I detect on Windows machines are detected because I realise that the antivirus software has silently shut down or stopped working because the virus got to it before it could raise the alarm. The other 10% are suspicious activity (network, disk, slowing down, alerts in software etc.).

1. Turn it off. This stops the problems spreading (to other machines and OTHER COMPANIES) or causing more damage (because, incidentally, you have *no* idea what this program is doing or not doing, because you can't even find out where the binary for it is called from!).

2. Remove the hard drive, put into a known-good machine (make sure it DOES NOT run anything from the drive, ideally under a Knoppix-like environment) compare the filesystem against a clean install of the same (obsolete?) operating system. This will usually show you exactly where the program is being called from and can be as simple as using diff. Because this is an old working, production machine, that diff will be HUGE, but you'll probably spot anything suspicious in the usual configuration directories. But it has to be done OFFLINE or you're wasting your time (otherwise you are basically asking the rogue program if it can see the rogue program you're looking for!).

3. If that turns up nothing unusual (I'm almost certain it would if you had caught anything, but you can't be sure), then you may have to look for out-of-place binary differences in the drive data (not just the filesystem), because it could be in the boot sector, in unused sectors, etc. This is now the territory of forensics and probably out of most people's depths (I would *theoretically* be able to do this, but I'm not sure I'd ever bother even with a £1,000,000 system - I'd be much more concerned with just eliminating the damn thing entirely by starting with fresh media, a fresh distribution and fresh source).

DO NOT... DO NOT... DO NOT... think that you've "found" the problem and just clear it up enough that you can carry on working with the same PC with the same distro, same data, etc. (which is what it sounds like you want to do). You have *no* idea what the program could have done to the system, at all, where else it's hidden itself or when it will pop back up again. DO NOT follow the usual Windows tech method of "Well, the antivirus detected and cleaned something, so the computer is perfectly clean, let's put it back in our network".

This computer has had bytes written to the disk, executable bytes, that neither you nor anything who has authorised access to the PC put there. You don't know where, why, how many or what they do, but they are running all the time you have that machine on with (presumbly, potentially and quite probably) complete machine privileges - they are potentially running at a level "higher" than any administrator user account the system has. Anything that has happened since the first point those bytes were present should be considered compromised until proven otherwise (lack of evidence is not proof that nothing happened!). You don't know if your backups are compromised. You don't know if the data is compromised. You don't know if the system is compromised. You don't know if any traffic that's *ever* passed through that system is compromised. You don't know if that system is being used to compromise other machines without your knowledge (salesman's laptops, external third-parties, other internal servers).

FFS.. just turn it off and worry about the HOW later. Consider the entire machine and any data that ever passed through it from your very first suspicion to be compromised. Yes, it's scary. No, you don't want to have to do it. Yes, the system will have to be brought down, a clean replacement built, etc. and it might be nothing more than an internal function to a well-known program. Yes, people are going to be unhappy. But customers and the business are going to be even more unhappy if you find that every copy of the every database in the company is compromised, you have no backups, no way to clean data and someone else is in control of your systems with *NO* way to replace them on any reasonable timescale.

Imagine you're the president and you're a target for assassination. You've just given your bomb-proof car to a random stranger on the street that you don't know, never met, have no way of finding out who it is, or what they have done to the car. You don't know them, your bodyguards don't know them, nobody knows them. They've had the car every night for a year and given it back to you every morning. You're driving that car EVERY SINGLE DAY of the year in between the stranger having it.

At times, something starts ticking somewhere in the car but you can't find the noise. You poke around in the back seat but don't find anything. You decide to leave it for several days/weeks and each day it starts ticking again. You walk past a garage and ask a mechanic about it (you've left the car with your wife and kids to take them shopping). The mechanic (and several other people qualified in their field who work at the garage) tells you to have the car taken off the road and looked at properly because they can't tell what the noise is just by your description but they and you both agree that it could be a bomb. You nod along sagely, but keep driving it.

You decide to run it in for a few days more, get some more people's opinions about the noise, but never actually letting anyone have a look at it. The mechanics go out of their way to hound you through the next couple of days into getting it serviced (not by them necessarily, because they have nothing invested in whether your car blows up or not, but they are just concerned for you) or at the very least stop using the damn thing.

That's what you've just done with that machine.

OK guys, calm down...

ledow, take a breath, a drink and maybe a valium... If you get this worked up by someone being interested in a different aspect of the problem than you have, you must find life to be unbearable.

inSpawn, I would say similar to you except you are don't seem to be as hyper and explosive...

2 quick things and then I'll drop it since:
1) This is not my system and I have already suggested rebuilding it with something newer and more current, or just reinstalling the original system and not worrying about upgrading it... It's not my decision to make and ledow can stick his "FFS" someplace with his assumptions that it's up to me and I have the decision power here.

2) You two don't find it of any interest that the problem only affects the services that start after the system leaves interactive and only starts non-interactive, then fine, you have both made you perspective QUITE apparent and have offered excellent information, the majority of which I am well aware of.

Now if you two will quit shoving your perspective down my and everyone's throat and let me see if anyone else finds it of interest and might have something to say.

We now know that it is an infection and some of us may find the how as of as much interest as the what... OK?

Before I go into things, you actually being relatively new here at LQ in terms of interaction with the community, I remind you the Linux Security forum exists for one and one reason only: to provide clarity in all security and related issues concerning GNU/Linux. The only thing this forum is concerned with is facts. That means there is no room for unfounded accusations, ego tripping, FUD, unbridled speculation or OT discussions. To allow your fellow LQ members to work with you on security issues you are required to post factual information.

So, while I agree Ledow goes off the deep end a wee bit with analogies even I find, ahhh, peculiar, you still only talk about things instead of posting any "evidence" or leads. Since LQ already shares knowledge and provides guidance (and free of charge and in our members spare time I might add), you must understand that only your level of willingness to cooperate and focus on the facts is required to make the outcome be succesful.


So let's turn this thread back on topic and ask for results of the things you said you've already done or are still doing like the outcome of filesystem verification, login record and log anomaly checks and such. With you vast UNIX experience it should be trivial for you to produce audit results.

It may be normal for the service you mentioned to start up. Attackers will modify services that are supposed to run normally.

If this is a Fedora (or SyS V init system based like most except debian & slackware) then you would use "chkconfig" to disable services. If the link to the startup/shutdown scripts don't disappear & it isn't an xinetd launched service, to me this would be further evidence of foul play. I don't mean to imply that if "sudo /sbin/chkconfig swap off" does behave as expected, that things are hunky dorey however. At best, you will be eliminating the annoying message. Also grep your startup scripts for 'swapd'.

It doesn't seem right that a swapd program exists in two locations. /usr/bin/ is definitely not where you would expect a service to exist.

Running "netstat --program" may provide clues as well.