LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices


Reply
  Search this Thread
Old 06-25-2008, 11:57 AM   #1
lionking_x
Member
 
Registered: Aug 2003
Posts: 36

Rep: Reputation: 15
Linux Patching Ideas


Consider this scenario, you have 100 Linux servers, both SuSE, 8, 9 10 and RHEL 3,4.

What are good strategies for patching these servers. These servers are not being patched regularly and I was looking for ideas on how to do this. We have tried ZenWorks, but that just lists a zillion patches and it is very difficult to decide which ones to apply, cannot trust Novell on that and it takes countless hours doing it. Some of our production servers are running on SLES 9 and are not patched, this worres me.

What do you do at your organization to keep your Linux servers patched, how do you determine the urgency of a patch/patches.

Thanks for all ideas
lk
 
Old 06-25-2008, 12:53 PM   #2
farslayer
LQ Guru
 
Registered: Oct 2005
Location: Northeast Ohio
Distribution: linuxdebian
Posts: 7,247
Blog Entries: 5

Rep: Reputation: 191Reputation: 191
Have you looked at puppet ?
 
Old 06-25-2008, 02:28 PM   #3
salasi
Senior Member
 
Registered: Jul 2007
Location: Directly above centre of the earth, UK
Distribution: SuSE, plus some hopping
Posts: 4,070

Rep: Reputation: 897Reputation: 897Reputation: 897Reputation: 897Reputation: 897Reputation: 897Reputation: 897
There are various strategies that you could apply. Broadly (and with slightly flexible and overlapping categories):
  1. Everything must be always as up-to-date as practicable
  2. All security patches must always be applied, plus significant others
  3. Quantified risk (by service)
  4. Separate servers into categories (high, med and low risk) and have policies for each category
  5. Hide under the desk and wear a tinfoil hat. And moose antlers. While singing 'I'm walking backwards to Christmas'. Backwards.
And it is possible to combine these (but probably not the last one, if you want any of the others to work).

Now (this next statement is going to sound unpleasant and unduly agressive at first reading....) if you were working in a really professional organisation with, say, ISO 9001/QS 900 cert, you'd probably have to have a risk assesment and and some kind of procedure that says what you have to do as a result of this risk assesment.

It might still be worth doing this, maybe in a slightly less formal way, even if you don't get audited every six months (and, even if you do and the auditors haven't asked to see it, it might still be worth doing it and hiding it {not going out of your way to make it available} to the auditors).

So it might be rational to do something like say that all machines that provide perimeter security are in the highest security class and for those (plus a select few others) there will be a daily check for security updates and those will be tested/applied within the next working day. Plus any obviously relevant other stuff, but with a longer period.

The next class might be non-perimeter but 'embarassing' info and that might have a different set of criteria and timescales. Or machines for which significant productivity or revenue is lost when they are not funtioning.

And then the rest with only internal D-O-S possibilities but no critical info might be more relaxed still, particularly if they only provide internal services of a non-urgent nature.

Trouble is, you'll probably find that you've got a non-supported machine or two in the highest risk category. Are you prepared to find that out, because that could be troubling? And work creating.

And you'll probably need a few 'trial servers' that you can test patches on. And with this array of different specs of machines, that might not be the low number (oh, say, 1) that you would really like. And while that might have the effect of driving you to rationalise your array of server software and configurations, it will be an inconvenience.

(And while you are there with categories and procedures, you would probably want to check whether backup frequency corresponds with data criticallity.)

Now the problem with all this should be becoming clear. How do you do this without being seen as a troublemaker/person with axe to grind for massive budgets/person seeking promotion by claiming that no one else knows how to run anything, ever/generally a pita? Did you used to like your job?
 
Old 06-25-2008, 02:57 PM   #4
lazlow
Senior Member
 
Registered: Jan 2006
Posts: 4,363

Rep: Reputation: 172Reputation: 172
If at all possible I would move them all to the same distro and version. This dramatically reduces the complexity of the problem. You can then create your only local repo filled with tested/verified updates(also reducing you download bandwidth). When new security threats arise you only have to look for one solution rather than the 5 (?) different solutions you currently have. This also simplifies updating. You can have one machine updated with the unupdated machine held in reserve. If everything checks out, the old server gets updated and you move to the next machine. If something fails, you just fire up the reserve machine, and start troubleshooting the updated machine.

Salasi has a lot of good points too.
 
Old 06-25-2008, 03:43 PM   #5
culaterout
Member
 
Registered: Jul 2006
Location: colorado
Distribution: Debian, Arch Linux, Linux Mint, Ubuntu, Fedora, Suse, Mepis, Redhat, Sayabon, mandrake and android (
Posts: 192

Rep: Reputation: 29
First,

I like most of the ideas here...

What type of company is this?

4) On a Personnel note: I don't care for Novell Suse it sucks lots of package updates to download... Not enough control at the Command line loaded in... Have to download far to many command line programs or commands... Don't forget zypper!!

(Your Boss and others like GUI to see what your talking about.. Pictures worth a thousand words...)


Sounds like your approaching this from a Outside threat.

What about firewalls, proxy servers and cisco routers....

Other forms of security and control...

Still your biggest threat is a User...

Look at this link..

http://forums.serverbeach.com/archiv....php/t-44.html

Good Luck...
 
Old 06-25-2008, 06:17 PM   #6
chrism01
LQ Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 7.7 (?), Centos 8.1
Posts: 17,735

Rep: Reputation: 2523Reputation: 2523Reputation: 2523Reputation: 2523Reputation: 2523Reputation: 2523Reputation: 2523Reputation: 2523Reputation: 2523Reputation: 2523Reputation: 2523
Ideally, go with lazlow post #4. You should pick a distro/ver and go with it. With RH I believe you can designate a local update server to hold/distribute updates.
You should also have an 'update test' box ie clone the prod boxes one at a time to a test box and try the updates if you are worried, before they go to prod systems.

An intermediate situation would be to at least reduce your multiple versions ie to update all your RH systems to the latest RH, and all your Suse systems to the latest Suse.
Your management should understand the need to do that at least.

Later on, you can consolidate to one distro.

What you need to do is write a nice doc for the management, explaining the issue and what it costs(!) (financially and security-wise) to even try and maintain all those systems.
Don't forget to try and estimate the financial costs if a security breach/patch failure occurs.
Money talks....
 
Old 06-26-2008, 10:08 AM   #7
salasi
Senior Member
 
Registered: Jul 2007
Location: Directly above centre of the earth, UK
Distribution: SuSE, plus some hopping
Posts: 4,070

Rep: Reputation: 897Reputation: 897Reputation: 897Reputation: 897Reputation: 897Reputation: 897Reputation: 897
Quote:
Originally Posted by lazlow View Post
If at all possible I would move them all to the same distro and version. This dramatically reduces the complexity of the problem.
This comment is absolutely correct, but:
  1. I'm assuming that you have apps and configurations that you have to test when installed on each distro, and the re-testing is a pain and that this drives you to keeping older distros longer than you would ideally like.
  2. I'm also assuming that there was a good reason for choosing to use both RH and SuSE, like the availability of certain apps for one or the other.
  3. I'm also assuming that you have only got to this rather undesirable place because your development process is somewhat (a bit, a lot?) control. That is, you have several small groups doing development and they all go their own way on everything, irrespective of what has been done to bring order to the process (maybe because management, in that restricted sense, is ineffectual).

The professional way to look at this is to asses everything and then look at the various advantages and disadvantages (costs). I would expect that one of the outcomes of this process is a knowledge of how much, in various ways, the current process is costing you (if only in risk) and that would drive you towards simplification.

On the other hand, I also have a certain amount of time for someone who can look at this and say 'that's ridiculous, we have to simplify'. It does save a lot of work.

I doubt that you will get down to the ideal situation of just one version of one distro, but I'm sure you could get closer, with a bit of co-operation.

While you are there, you probably also ought to thinking about server consolidation and/or virtualisation to reduce the amount of hardware and the power consumption at the same time as reducing the ongoing workload. (Note that most 'unsophisticated' organisations have collections of hardware that grow year-on-year until either they run out of room, their air conditioning hardware can't cope, or they can't supply power to it any more. That there isn't a more sophisticated process in place seems to me to be tragic, but then that's me.)

You should also remember that, in any change process, the human and interpersonal factors are probably a bigger/more knotty problem than the purely technical ones.
 
Old 06-27-2008, 11:24 AM   #8
lionking_x
Member
 
Registered: Aug 2003
Posts: 36

Original Poster
Rep: Reputation: 15
Thanks for the many replies. The problem at the place I work is resources. They do not have enough admins to do that task. Couple of guys managing and doing everything from configuring webservers/applications servers/database servers etc. There is hardly much time to look at patching, and getting resources here is not easy, so I am looking at what best I could do. I might get an intern to help me soon.

Earlier this year, we went ahead and boldly upgraded a SuSE Linux version 9 to 10.1. That broke the apache, which would not start and gave some library errors. We do not have a proper backup system in place for the linux systems, something like a Ghost in Windows. It becomes incredibly hard to revert back. We have been looking at Novell's Zenworks which is promising, but is very tedious and a full time job.

Talking about backups, any of you folks have any tools to take a full ghost image of a linux server ? which I can revert back pretty easily if a major kernel patch goes awry ?

How about Levanta ? they have their own tool which could help with all this.

Thanks
 
Old 06-27-2008, 11:47 AM   #9
lionking_x
Member
 
Registered: Aug 2003
Posts: 36

Original Poster
Rep: Reputation: 15
Yes, consolidating into one distro is certainly beneficial. That will be one of our future projects.


Chrism01, you said "You should also have an 'update test' box ie clone the prod boxes one at a time to a test box and try the updates if you are worried, before they go to prod systems."

How do yo suggest we clone a prod box ? I can tar stuff, but how do I clone/ghost the entire system ? What have the rest of you been using ?

Thanks
 
Old 06-28-2008, 04:37 AM   #10
salasi
Senior Member
 
Registered: Jul 2007
Location: Directly above centre of the earth, UK
Distribution: SuSE, plus some hopping
Posts: 4,070

Rep: Reputation: 897Reputation: 897Reputation: 897Reputation: 897Reputation: 897Reputation: 897Reputation: 897
Quote:
Originally Posted by lionking_x View Post
The problem at the place I work is resources. They do not have enough admins to do that task. Couple of guys managing and doing everything from configuring webservers/applications servers/database servers etc. There is hardly much time to look at patching, and getting resources here is not easy, so I am looking at what best I could do. I might get an intern to help me soon.
...oddly, that was what I suspected from your original post

Quote:
Earlier this year, we went ahead and boldly upgraded a SuSE Linux version 9 to 10.1. That broke the apache, which would not start and gave some library errors.
This is always a risk; and making a big jump over several versions (or to a different distro) only makes the risk bigger. And that's why you can't just say 'its bound to work, if its got Apache' (or whatever).

Quote:
We do not have a proper backup system in place for the linux systems, something like a Ghost in Windows. It becomes incredibly hard to revert back. We have been looking at Novell's Zenworks which is promising, but is very tedious and a full time job.

Talking about backups, any of you folks have any tools to take a full ghost image of a linux server ? which I can revert back pretty easily if a major kernel patch goes awry ?
I'm assuming that you do have some kind of backup system that does a good job (well, adequate, anyway) of backing up anything that you create yourself (whether that be database files, alteration to config files, etc) but which doesn't backup everything and doesn't give you a 'trivial re-install'.
If you don't have this minimal backup system, cure this now (or sooner).
And, I'm also guessing that you have a cupboard full of media without much order and that you can't find a particular one quickly either. Or tell what you have and what has gone missing and is currently on someones desk waiting for something to happen. If so, this is bad and will get you sooner or later.

Err, given the nature of your org, I'm also assuming that you may not have tested your restore procedure. This is pretty urgent. I'm also assuming you don't do media testing, to check that your media are still in good condition after a time period. Depending on the media that you use, this can be an issue too.

My preference is to split the problem into two parts; the 're-install distro software' part (with or without patches) and the 'restore your own ip' part. The reason for this is that I have a limited confidence in the 'restore from a ghost image' part if you don't do it on absolutely identical hardware.

My guess is that this kind of org isn't good at standardising hardware, and even if you did standardise at some instantaneous snapshot as time passes you will inevitably accumulate different hardware. This means that guaranteeing that you will have identical hardware to install onto is difficult or maybe even impossible.

For the re-install distro software part:
  • it's obviously easier with few versions of one distro
  • you could do a standard 'base install' plus documented add-ons
For your own ip part
  • don't forget conf files on anything you are likely to configure yourself
  • for desktop machines, getting the /home tree covers most of everything; for you it probably doesn't, you'll need some of /etc and it may be easiest to grab all of /etc
  • with rpm-based distros look at storing the output of rpm-qa as one of your bits of documentation, probably with lshw, a sample dmesg and sample system logs
  • I think there is a strong case for putting a little file which documents the initial build time & date into a standardised place. Then anything with a later date and time (mtime) than that needs to be considered for backup, because either you've modified it or its a patch you've installed (note that this automatically doesn't get everything; 'restores' may be dated earlier)

One of the things about all of this is organisation; if you fall into it, it will be a bit of a mess and hard work. If you automate some of the work it can get more manageable, even if you are doing more, because the standard parts can be automated. Even a spreadsheet (or a database or python script) that documents the install options on a machine will be a help.

On the other hand, expect the argument 'we don't have the time to get organised'.
 
Old 06-28-2008, 07:27 AM   #11
culaterout
Member
 
Registered: Jul 2006
Location: colorado
Distribution: Debian, Arch Linux, Linux Mint, Ubuntu, Fedora, Suse, Mepis, Redhat, Sayabon, mandrake and android (
Posts: 192

Rep: Reputation: 29
Quote:
Originally Posted by lionking_x View Post
Thanks for the many replies. The problem at the place I work is resources. They do not have enough admins to do that task. Couple of guys managing and doing everything from configuring webservers/applications servers/database servers etc. There is hardly much time to look at patching, and getting resources here is not easy, so I am looking at what best I could do. I might get an intern to help me soon.

Thanks


Well it sounds like your probably in too many meetings how to resolve issues...

People pulling at you one way or the other....

I hope corporate doesn't visit you....

You have no backup plain? What about a power loss? What do you do? I hope these severs are running on a UPS(Uninterruptable Power Supply)...?

Your in real trouble patching holes to keep the boat afloat....

After reading your first post again decided this was a game...

100 servers you wouldn't even been in general location of a 100 servers..You would have to tunnel into that many servers.... Second off even in the hospital I worked at we only had maybe 10 servers for 10,000 users....

Worked in DC not even there unless I was at the pentagon.. Wouldn't I have access to that many servers... Causes a conflict to much control over a system..

Lol,


100 servers pls.....

Last edited by culaterout; 06-29-2008 at 01:25 AM.
 
Old 07-01-2008, 03:23 PM   #12
lionking_x
Member
 
Registered: Aug 2003
Posts: 36

Original Poster
Rep: Reputation: 15
Salasi

Quote:
I'm assuming that you do have some kind of backup system that does a good job (well, adequate, anyway) of backing up anything that you create yourself (whether that be database files, alteration to config files, etc) but which doesn't backup everything and doesn't give you a 'trivial re-install'.
If you don't have this minimal backup system, cure this now (or sooner).
And, I'm also guessing that you have a cupboard full of media without much order and that you can't find a particular one quickly either. Or tell what you have and what has gone missing and is currently on someones desk waiting for something to happen. If so, this is bad and will get you sooner or later.

Err, given the nature of your org, I'm also assuming that you may not have tested your restore procedure. This is pretty urgent. I'm also assuming you don't do media testing, to check that your media are still in good condition after a time period. Depending on the media that you use, this can be an issue too.
The answer is Yes and No, we do have a decent backup system. Everything gets backed onto tape. All database files, data files, config files etc. The only issue we have is backing up of the entire linux server, which we are not doing. Everything has raid-1 for more protection, pretty much standard server class hardware. If one of the linux systems crashes, and we cannot bring it back up,we rebuild linux on the box, install the application/database what ever and are back in business. This takes time, but for the past 3 years, it has never happened. Not a single crash, can you believe it ? The filesystem crashes which did happen, we could recover my simple fsck's. IS IT NECESSARY TO BACKUP EVERY PARTIOTION ON EVERY LINUX SERVER WE HAVE ????

Quote:
One of the things about all of this is organisation; if you fall into it, it will be a bit of a mess and hard work. If you automate some of the work it can get more manageable, even if you are doing more, because the standard parts can be automated. Even a spreadsheet (or a database or python script) that documents the install options on a machine will be a help.
We do have a standard document, try to follow some basic steps. But with different distros and different app/database requirements it gets messy.
We have cfengine for lots of admin work stuff, but again, when fewer people try to do too many things, quality suffers a little.

Quote:
On the other hand, expect the argument 'we don't have the time to get organised'.
Why does this surprise you ? these places are cutting costs all the time and do not seem to care much. The place I work is not a Private organization (thats as much I can say). There is lots of beurocracy. Things are slow. But the requests are typical, they need everything done yesterday. Thanks for the tip though.

I like your suggestion, but am looking at a centralized patch management console kinda thing (similar to zenworks) to start real work on this. This is the only way we can get some order, I would imagine ?
Does this sound right ? Fortunately, our network team is rock solid and we have everything tied down very well. Most of our apps are internal, which is another good thing.

Thanks
 
Old 07-02-2008, 09:56 AM   #13
lionking_x
Member
 
Registered: Aug 2003
Posts: 36

Original Poster
Rep: Reputation: 15
Quote:
Well it sounds like your probably in too many meetings how to resolve issues...
People pulling at you one way or the other....
I hope corporate doesn't visit you....
I was patching here and there, for critical fixes, like ssh, samba, but have not covered everything as we should have. Cor-porate/audit folks did visit us, despite we being behind on patches they could not penetrate the network and found us to be pretty solid. This could be because of the fronting network infrastructure we have in place, you know the firewalls, cisco pix etc. So that makes my argument of getting everything uptodate even weaker.

Quote:
You have no backup plain? What about a power loss? What do you do? I hope these severs are running on a UPS(Uninterruptable Power Supply)...?
As I explained earlier, we backup all config files, all databases, all data related reports, svn etc. What we do not have are ghost images of the linux servers we have, or even tars / basic dd zips.

Quote:
Your in real trouble patching holes to keep the boat afloat....
This is probably true. That is the reason, we are working aggressively to find a strategy to go with. Still evaluating the zenworks and the you server options and ofcourse an extra hand to help. Fortunately most of the applications served are for internal use, this will change in a couple of years and we would like to be prepared and ready for that.

Quote:
After reading your first post again decided this was a game...
100 servers you wouldn't even been in general location of a 100 servers..You would have to tunnel into that many servers.... Second off even in the hospital I worked at we only had maybe 10 servers for 10,000 users....
100servers sit on just 3 racks, that is like 2 cubicles or a little more.
We have ILO setup for everything so you have hardware access to all systems. Not sure what the hospital does, but the place I work which is not private, has big plans for the future. At present they are doing all this development and soon it will go external. If I ring
firealarm, things would happen right away, but am still contemplating doing that.

Why 100 servers ? This is a good question, I myself think that we could have around 75 and would have been enough. But that is the way it is, we have the entire IBM's edge suite, a whole bunch of weblogic/websphere app servers, database servers, etl servers etc.

QUOTE]
100 servers pls.....[
[/QUOTE]

Dude, you seem to have some experience with this. Can you give some more scenarios on how we can accomplish this, by automating a lot of the process.
 
Old 07-02-2008, 03:57 PM   #14
Tinkster
Moderator
 
Registered: Apr 2002
Location: in a fallen world
Distribution: slackware by choice, others too :} ... android.
Posts: 23,067
Blog Entries: 11

Rep: Reputation: 914Reputation: 914Reputation: 914Reputation: 914Reputation: 914Reputation: 914Reputation: 914Reputation: 914
Shuffled about for better exposure and cross-forum linkage ...
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
LXer: Ten ideas about Ideas LXer Syndicated Linux News 0 11-05-2006 08:21 AM
Linux security patching? dwarf007 Linux - General 4 04-07-2006 08:32 PM
Patching in linux..... westman Linux - Newbie 3 02-20-2006 11:17 PM
HELP on GFS - patching linux kernel antonioxcom Linux - General 1 07-06-2004 01:20 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Server

All times are GMT -5. The time now is 12:30 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration